zephyr

zephyr copied to clipboard

Namespaced the generated headers with zephyr/ to prevent potential conflict with other headers.

Introduced a Kconfig (CONFIG_LEGACY_GENERATED_INCLUDE_PATH) to support legacy generated header include paths in the mean time.

Fixes #60377 Fixes #68035

• [x] Missing changes to the 2 repos that depend on this are required in order to merge the PR
  • sof
    • https://github.com/thesofproject/sof/pull/8459
    • https://github.com/zephyrproject-rtos/sof/pull/34
  • percepio
    • https://github.com/percepio/percepio/pull/5
    • https://github.com/percepio/percepio/pull/6
    • https://github.com/zephyrproject-rtos/percepio/pull/10
• [x] Missing a entry to the migration document

Automation script:

from pathlib import Path
import re

EXTENSIONS = ("c", "h", "cpp", "rst", "md", "S", "ld", "py", "svg")

HEADERS = [
    "app_version",
    "cmake_intdef",
    "core-isa-dM",
    "devicetree_generated",
    "driver-validation",
    "kobj-types-enum",
    "linker-kobject-prebuilt-data",
    "mcuboot_version",
    "offsets",
    "otype-to-size",
    "otype-to-str",
    "strerror_table",
    "strsignal_table",
    "syscall_list",
    "version",
    "zsr",
]

HEADERS_H = [hdr + ".h" for hdr in HEADERS]

for p in Path(".").glob("zephyr/**/*"):
    if not p.is_file() or p.suffix and p.suffix[1:] not in EXTENSIONS:
        continue

    # skip git & github folders
    if (str(p).startswith("zephyr/.git")):
        continue

    # skip release docs
    if (str(p).startswith("zephyr/doc/releases")):
        continue

    content = ""
    with open(p, 'r', newline='') as f:
        for line in f:
            # for lines in sources
            m = re.match(r"^(.*)#include <(.*)>(.*)$", line)
            # for lines in docs
            n = re.match(r"^(.*)include/generated/(.*)\.h(.*)$", line)
            if (m and (m.group(2).startswith("syscalls/") or m.group(2) in HEADERS_H)):
                content += (m.group(1) + "#include <zephyr/" +
                            m.group(2) + ">" + m.group(3) + "\n")
            elif (n and n.group(2) in HEADERS):
                content += (n.group(1) + "include/generated/zephyr/" +
                            n.group(2) + ".h" + n.group(3) + "\n")
            else:
                content += line

    with open(p, "w", newline='') as f:
        f.write(content)

Requirements for Treewide Changes

• [x] The zephyr repository must apply the ‘treewide’ GitHub label to any issues or pull requests that are treewide changes
• [x] The person proposing a treewide change must create an RFC issue describing the change, its rationale and impact, etc. before any pull requests related to the change can be merged
• [x] The project’s Architecture Working Group (WG) must include the issue on the agenda and discuss whether the project will accept or reject the change before any pull requests related to the change can be merged (with escalation to the TSC if consensus is not reached at the WG)
• [ ] The Architecture WG must specify the procedure for merging any PRs associated with each individual treewide change, including any required approvals for pull requests affecting specific subsystems or extra review time requirements
• [ ] The person proposing a treewide change must email [email protected] about the RFC if it is accepted by the Architecture WG before any pull requests related to the change can be merged

this PR is getting more involved than I initially expected

EDIT: And CI is complaining about some lines that are not modified in this PR:

Run if [[ ! -s "compliance.xml" ]]; then
Error: See https://www.pylint.org/ for more details

 C03[25](https://github.com/zephyrproject-rtos/zephyr/actions/runs/6542835611/job/17766473397?pr=63973#step:9:26):Unnecessary parens after '=' keyword (superfluous-parens)
File:scripts/kconfig/kconfiglib.py
Line:2931
Column:0
 C0325:Unnecessary parens after '=' keyword (superfluous-parens)
File:scripts/kconfig/kconfiglib.py
Line:4310
Column:0
 C0325:Unnecessary parens after '=' keyword (superfluous-parens)
File:scripts/kconfig/kconfiglib.py
Line:4448
Column:0
Error: Process completed with exit code 1.

https://github.com/zephyrproject-rtos/zephyr/blob/b272760f062b36fe6ce5d14c7a3d9065d40a570a/scripts/kconfig/kconfiglib.py#L2931 https://github.com/zephyrproject-rtos/zephyr/blob/b272760f062b36fe6ce5d14c7a3d9065d40a570a/scripts/kconfig/kconfiglib.py#L4310 https://github.com/zephyrproject-rtos/zephyr/blob/b272760f062b36fe6ce5d14c7a3d9065d40a570a/scripts/kconfig/kconfiglib.py#L4448

This is fixed in the last commit

@ycsin hey thanks for putting this together, since it's a major change I added the treewide and architecture review tags so this can be discussed in the architecture WG (cc @carlescufi)

Rebased on top of main Moved LEGACY_GENERATED_INCLUDE_PATH into the Compatibility menu

---

EDIT:

usb_dc_native_posix_adapt.c fails to build after the rebase

[1006/1063] Building C object zephyr/drivers/usb/device/CMakeFiles/drivers__usb__device.dir/usb_dc_native_posix_adapt.c.obj
FAILED: zephyr/drivers/usb/device/CMakeFiles/drivers__usb__device.dir/usb_dc_native_posix_adapt.c.obj 
ccache /usr/bin/gcc -DKERNEL -D_POSIX_C_SOURCE=200809 -D_XOPEN_SOURCE=600 -D_XOPEN_SOURCE_EXTENDED -D__LINUX_ERRNO_EXTENSIONS__ -D__ZEPHYR_SUPERVISOR__ -D__ZEPHYR__=1 -I/__w/zephyr/zephyr/twister-out/native_sim_64/samples/subsys/shell/shell_module/sample.shell.shell_module.usb/zephyr/include/generated/zephyr -I/__w/zephyr/zephyr/include -I/__w/zephyr/zephyr/twister-out/native_sim_64/samples/subsys/shell/shell_module/sample.shell.shell_module.usb/zephyr/include/generated -I/__w/zephyr/zephyr/soc/posix/inf_clock -I/__w/zephyr/zephyr/boards/posix/native_sim -I/__w/zephyr/zephyr/lib/posix/getopt/. -I/__w/zephyr/zephyr/scripts/native_simulator/common/src/include -I/__w/zephyr/zephyr/scripts/native_simulator/native/src/include -I/__w/zephyr/zephyr/subsys/usb/device -isystem /__w/zephyr/zephyr/twister-out/native_sim_64/samples/subsys/shell/shell_module/sample.shell.shell_module.usb/modules/picolibc/picolibc/include -Wshadow -fno-strict-aliasing -Werror -Os -imacros /__w/zephyr/zephyr/twister-out/native_sim_64/samples/subsys/shell/shell_module/sample.shell.shell_module.usb/zephyr/include/generated/zephyr/autoconf.h -fno-printf-return-value -fno-common -g -gdwarf-4 -fdiagnostics-color=always -Wall -Wformat -Wformat-security -Wno-format-zero-length -Wno-pointer-sign -Wpointer-arith -Wexpansion-to-defined -Wno-unused-but-set-variable -Werror=implicit-int -fno-pic -fno-pie -fno-asynchronous-unwind-tables -fno-reorder-functions --param=min-pagesize=0 -fno-defer-pop -fmacro-prefix-map=/__w/zephyr/zephyr/samples/subsys/shell/shell_module=CMAKE_SOURCE_DIR -fmacro-prefix-map=/__w/zephyr/zephyr=ZEPHYR_BASE -fmacro-prefix-map=/__w/zephyr=WEST_TOPDIR -ffunction-sections -fdata-sections -m64 -fPIC -fvisibility=hidden -nostdinc -isystem /usr/lib/gcc/x86_64-linux-gnu/11/include -ffreestanding -fno-builtin -D_POSIX_THREADS -D_POSIX_C_SOURCE=200809 -std=c11 -MD -MT zephyr/drivers/usb/device/CMakeFiles/drivers__usb__device.dir/usb_dc_native_posix_adapt.c.obj -MF zephyr/drivers/usb/device/CMakeFiles/drivers__usb__device.dir/usb_dc_native_posix_adapt.c.obj.d -o zephyr/drivers/usb/device/CMakeFiles/drivers__usb__device.dir/usb_dc_native_posix_adapt.c.obj -c /__w/zephyr/zephyr/drivers/usb/device/usb_dc_native_posix_adapt.c
/__w/zephyr/zephyr/drivers/usb/device/usb_dc_native_posix_adapt.c:21:10: fatal error: sys/socket.h: No such file or directory
   21 | #include <sys/socket.h>
      |          ^~~~~~~~~~~~~~
compilation terminated.

Created #64227 as it is failing on main as well

Another compilation issue getting fixed @ #64251

Ran this again the push CI job in the testing repo (https://github.com/zephyrproject-rtos/zephyr-testing/runs/18077860182) single failure but it's unrelated. Great job there, hopefully we can merge it soon, too bad we did not make to this at the last arch wg, we should probably do it early next time, I don't see any objections here anyway.

Needs a rebase btw.

rebased and remove modifications to scripts/kconfig/kconfiglib.py

rebased again for #64464

Architecture WG:

• Checked that we do not have an applicable Kconfig option anymore that can be used, instead we will need the new CONFIG_LEGACY_GENERATED_INCLUDE_PATH.
• Missing changes to the 2 repos that depend on this are required in order to merge the PR
• Missing a entry to the migration document

this PR is getting more involved than I initially expected

EDIT: And CI is complaining about some lines that are not modified in this PR:

@ycsin - those compliance issues could also be fixed in a separate PR. This one will require a respin anyway, so that might be a good option

rebased, created:

• https://github.com/thesofproject/sof/pull/8459
• https://github.com/percepio/percepio/pull/5
• https://github.com/percepio/percepio/pull/6

I'm not sure which category should this change falls into in the migration guide, any idea?

I'm not sure which category should this change falls into in the migration guide, any idea?

build system

@ycsin seems like this needs a rebase

This needs a rebase. Is the target milestone still v3.6.0?

This needs a rebase. Is the target milestone still v3.6.0?

I'm not sure tbh, does this need to go thru an architecture review?

I'm not sure tbh, does this need to go thru an architecture review?

Yes, it needs to follow the process described here: https://docs.zephyrproject.org/latest/contribute/guidelines.html#treewide-changes

Please do continue the work here, +1 from me on CI passing and arch wg agreement

This PR should be ready for the architecture review now

Testing the changes for autoconf.h with another commit.

List of files that currently includes the autoconf.h:

(.venv) ycsin@LAPTOP-ROG:~/zephyrproject$ find . -type f -exec grep -H '#include [<|"]autoconf\.h[>|"]' {} +
./modules/hal/stm32/scripts/tests/genllheaders/data/stm32_ll_tim.h:#include <autoconf.h>
./modules/hal/stm32/scripts/tests/genllheaders/data/stm32_ll_usart.h:#include <autoconf.h>
./modules/hal/stm32/scripts/genllheaders/header-template.j2:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_fmpi2c.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_lptim.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_wwdg.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_icache.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_gpio.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_pwr.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_adc.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_bdma.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_spi.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_utils.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_delayblock.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_dlyb.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_lpuart.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_opamp.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_ipcc.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_hsem.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_rcc.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_dmamux.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_dac.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_i3c.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_dma2d.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_i2c.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_system.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_comp.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_hrtim.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_swpmi.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_tim.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_crc.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_cortex.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_ucpd.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_cordic.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_dcache.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_sdmmc.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_lpgpio.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_fmac.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_usart.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_exti.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_rng.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_mdma.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_iwdg.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_dma.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_bus.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_crs.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_fmc.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_pka.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_rtc.h:#include <autoconf.h>
./modules/hal/stm32/stm32cube/common_ll/include/stm32_ll_fsmc.h:#include <autoconf.h>
./modules/hal/nxp/s32/mcux/devices/S32K344/S32K344_features.h:#include "autoconf.h"

Seems like it works this time, not sure what I missed last time for the autoconf.h changes. cc @carlescufi

rebased to fix conflicts

The PR branchs in percepio are now merged while this branch isn't, which is likely to cause havoc to percepio users?

EDIT: It probably won't because the west.yml isn't updated to use the newer version

Needs rebase

This pull request has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this pull request will automatically be closed in 14 days. Note, that you can always re-open a closed pull request at any time.

Removed Architecture Review as it had been reviewed a few months ago, it just requires approvals from the maintainers now.

Rebased and ran the script to update some new #include additions

@tejlmand Please revisit.

The following west manifest projects have been modified in this Pull Request:

Name	Old Revision	New Revision	Diff

Note: This message is automatically posted and updated by the Manifest GitHub Action.

Things are failing now because LEGACY_GENERATED_INCLUDE_PATH is now disabled by default (previously it was enabled by default).

Currently I'm just trying to see if hal_stm32 is the only one affected by the latest change, I think it probably makes sense to enable the LEGACY_GENERATED_INCLUDE_PATH by default so that this PR has less dependencies on the external modules, and disable + deprecate that after we have all PRs in external modules repo merged.

EDIT:

• [x] fixes for CI to pass (CONFIG_LEGACY_GENERATED_INCLUDE_PATH=n)
• [x] Remove last commit that patches the west.yml temporarily
• [x] Wait for CI to pass again (CONFIG_LEGACY_GENERATED_INCLUDE_PATH=y)

• [ ] wait for CI to pass (CONFIG_LEGACY_GENERATED_INCLUDE_PATH=n)

• [ ] Remove last commit (4b03a50bcd49ca4bffc2accfda7283b04f317371)

• [ ] Wait for CI to pass again (CONFIG_LEGACY_GENERATED_INCLUDE_PATH=y)

If CI passes with CONFIG_LEGACY_GENERATED_INCLUDE_PATH=n, why revert it?