Split heap-validate.c into smaller chunks. We have been adding all kind
of new APIs under this file and building it unconditionally whether
those APIs are needed/used or not. Many of those APIs have nothing to do
with the validation part.
Signed-off-by: Anas Nashif <anas.nashif@intel.com>
Since the retrieved heap memory statistics are identical to those of
the retrieved mem_block statistics, it makes sense to use a single
generic-named structure for both instead of two identical structures
with different names.
Signed-off-by: Peter Mitsis <peter.mitsis@intel.com>
In order to bring consistency in-tree, migrate all lib code to the new
prefix <zephyr/...>. Note that the conversion has been scripted, refer
to zephyrproject-rtos#45388 for more details.
Signed-off-by: Gerard Marull-Paretas <gerard.marull@nordicsemi.no>
Besides the current allocated/free bytes, keep track of
the maximum allocated bytes to help determine the heap
size requirements. Also, provide a function to reset
the statistic.
Signed-off-by: Damian Krolik <damian.krolik@nordicsemi.no>
add functions to get the sys_heap runtime statistics,
include total free bytes, total allocated bytes.
Signed-off-by: Chen Peng1 <peng1.chen@intel.com>
With 64 bytes heap and 1 byte allocation on a big heap, we get:
0 1 2 3 4 5 6 7
| h | h | b | b | c | 1 | s | f |
where
- h: chunk0 header
- b: buckets in chunk0
- c: chunk header for the first allocation
- 1: chunk mem
- s: solo free header
- f: end marker / footer
max_chunkid() was returning h->end_chunk - min_chunk_size(h), which is
5 because min_chunk_size() on a big heap is 2. This works if you
don't have the solo free header at 6 and the heap is like:
0 1 2 3 4 5 6
| h | h | b | b | c | 1 | f |
max_chunkid() in this case gives you 6 - 2 = 4, which is the right
chunkid for the last chunk header.
This commit replaces max_chunkid() with h->end_chunk and "<=" (less
than or equal to) with "<" (less than), so that it always compares
against the end maker chunkid, but the code won't touch the end maker
itself.
Signed-off-by: Yasushi SHOJI <yashi@spacecubics.com>
heap* had several places missing final else statement in the
if else if construct. This commit adds else {} to comply with
coding guideline 15.7.
Signed-off-by: Jennifer Williams <jennifer.m.williams@intel.com>
The lib/os/ had several places missing final else
statement in the if else if construct. This commit adds
else {} or simple refactor to comply with coding guideline 15.7.
- cbprintf_complete.c
- cbprintf_nano.c
- heap-validate.c
- heap.c
- onoff.c
- p4wq.c
- sem.c
Also resolves the checkpatch issue of comments should align * on
each line.
Signed-off-by: Jennifer Williams <jennifer.m.williams@intel.com>
The if ... else if ... construct was missing the final else.
This commit refactors it to comply with coding guideline 15.7.
The logic is to check if used or free, and do not increment
for the reserved chunks (first/last) in the heap.
Signed-off-by: Jennifer Williams <jennifer.m.williams@intel.com>
The size_t usage, especially in struct z_heap_bucket made the heap
header almost 2x bigger than it needs to be on 64-bit systems.
This prompted me to clean up our type usage to make the code more
efficient and easier to understand. From now on:
- chunkid_t is for absolute chunk position measured in chunk units
- chunksz_t is for chunk sizes measured in chunk units
- size_t is for buffer sizes measured in bytes
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
The end marker chunk was represented by the len field of struct z_heap.
It is now renamed to end_chunk to make it more obvious what it is.
And while at it...
Given that it is used in size_too_big() to cap the allocation size
already, we no longer need to test the bucket index against the
biggest index possible derived from end_chunk in alloc_chunk(). The
corresponding bucket_idx() call is relatively expensive on some
architectures so avoiding it (turning it into a CHECK() instead) is
a good thing.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Turn sys_heap_dump() into sys_heap_print_info() to better reflect
what it actually does, and improve the information being printed.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Instead of limiting the excess split-off to sufficiently large chunks
in split_alloc(), let's allow normal allocations to create "solo free
headers" just like with aligned allocations. There is no point leaving
them in the allocated chunk if the user didn't ask for it. Doing so
makes them eligible for merging at the next opportunity and potentially
reusable sooner.
Also make the validation code aware of them.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
One fundamental validation criteria is to never have consecutive free
chunks. If that ever happens we failed to merge them. That means a free
chunk must always be surrounded by used chunks.
It is a pain to extend valid_chunk() with new rules as it is.
So a VALIDATE() macro is introduced to make things easier to work with.
It also allows for isolating each test, possibly making VALIDATE() into
__ASSERT() to determine exactly which test is tripping when debugging.
Finally, because of that new validation rule, sys_heap_validate() must
be modified so not to use valid_chunk() while it is flipping all the
"used" flags. So let's run valid_chunk() up front before alterating
chunk headers.
Now sys_heap_validate() has become justifiably more expensive and a few
emulated targets are about to bust the tests/lib/heap test timeout. So
bump the timeout as well.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
This struct is taking up most of the heap's constant footprint overhead.
We can easily get rid of the list_size member as it is mostly used to
determine if the list is empty, and that can be determined through
other means.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
We already have chunk #0 containing our struct z_heap and marked as
used. We can add a partial chunk at the very end that is also marked
as used. By doing so there is no longer a need for checking heap
boundaries at run time when merging/splitting chunks meaning fewer
conditionals in the code's hot path.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
It is possible to remove a few fields from struct z_heap, removing
some runtime indirections by doing so:
- The buf pointer is actually the same as the struct z_heap pointer
itself. So let's simply create chunk_buf() that perform a type
conversion. That type is also chunk_unit_t now rather than u64_t so
it can be defined based on CHUNK_UNIT.
- Replace the struct z_heap_bucket pointer by a zero-sized array at the
end of struct z_heap.
- Make chunk #0 into an actual chunk with its own header. This allows
for removing the chunk0 field and streamlining the code. This way
h->chunk0 becomes right_chunk(h, 0). This sets the table for further
simplifications to come.
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
Let's provide accessors for getting and setting every field to make the
chunk header layout abstracted away from the main code. Those are:
SIZE_AND_USED: chunk_used(), chunk_size(), set_chunk_used() and
chunk_size().
LEFT_SIZE: left_chunk() and set_left_chunk_size().
FREE_PREV: prev_free_chunk() and set_prev_free_chunk().
FREE_NEXT: next_free_chunk() and set_next_free_chunk().
To be consistent, the former chunk_set_used() is now set_chunk_used().
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
First, some renames to make accessors more explicit:
size() --> chunk_size()
used() --> chunk_used()
free_prev() --> prev_free_chunk()
free_next() --> next_free_chunk()
Then, the return type of chunk_size() is changed from chunkid_t to
size_t, and chunk_used() from chunkid_t to bool.
The left_size() accessor is used only once and can be easily substituted
by left_chunk(), so it is removed.
And in free_list_add() the variable b is renamed to bi so to be
consistent with usage in sys_heap_alloc().
Signed-off-by: Nicolas Pitre <npitre@baylibre.com>
The existing mem_pool implementation has been an endless source of
frustration. It's had alignment bugs, it's had racy behavior. It's
never been particularly fast. It's outrageously complicated to
configure statically. And while its fragmentation resistance and
overhead on small blocks is good, it's space efficiencey has always
been very poor due to the four-way buddy scheme.
This patch introduces sys_heap. It's a more or less conventional
segregated fit allocator with power-of-two buckets. It doesn't expose
its level structure to the user at all, simply taking an arbitrarily
aligned pointer to memory. It stores all metadata inside the heap
region. It allocates and frees by simple pointer and not block ID.
Static initialization is trivial, and runtime initialization is only a
few cycles to format and add one block to a list header.
It has excellent space efficiency. Chunks can be split arbitrarily in
8 byte units. Overhead is only four bytes per allocated chunk (eight
bytes for heaps >256kb or on 64 bit systems), plus a log2-sized array
of 2-word bucket headers. No coarse alignment restrictions on blocks,
they can be split and merged (in units of 8 bytes) arbitrarily.
It has good fragmentation resistance. Freed blocks are always
immediately merged with adjacent free blocks. Allocations are
attempted from a sample of the smallest bucket that might fit, falling
back rapidly to the smallest block guaranteed to fit. Split memory
remaining in the chunk is always returned immediately to the heap for
other allocation.
It has excellent performance with firmly bounded runtime. All
operations are constant time (though there is a search of the smallest
bucket that has a compile-time-configurable upper bound, setting this
to extreme values results in an effectively linear search of the
list), objectively fast (about a hundred instructions) and amenable to
locked operation. No more need for fragile lock relaxation trickery.
It also contains an extensive validation and stress test framework,
something that was sorely lacking in the previous implementation.
Note that sys_heap is not a compatible API with sys_mem_pool and
k_mem_pool. Partial wrappers for those (now-) legacy APIs will appear
later and a deprecation strategy needs to be chosen.
Signed-off-by: Andy Ross <andrew.j.ross@intel.com>
Hess Nathan
Anas Nashif
Peter Mitsis
Gerard Marull-Paretas
Damian Krolik
Nicolas Pitre
Chen Peng1
Yasushi SHOJI
Jennifer Williams
Kumar Gala
Andy Ross