pkg: USEPKG vs USEMODULE

[miri64]

There seem to be some confusion all throughout the code-base about the USEPKG macro and how to use it. My interpretation at this point: USEPKG pulls the package in (i.e. just downloads and patches it), USEMODULE then builds modules that are part of that module. Yes, USEPKG lets the buildsystem step into the package's path, but I see it more like updating DIRS in sys or net/gnrc. However, this does not seem to be the case with all packages. Some use USEPKG and USEMODULE interchangeably, or even require the user only to provide USEPKG, while failing to build if a user provides it in USEMODULE. I think my interpretation makes sense in terms of user-friendliness for several reasons:

• It provides a clear semantic between what a package (a mechanism for building and patching an external module) is and what a module (a build-unit provided to the RIOT image) is
• It allows a package to have sub-modules (see lwIP) or modules that have nothing to do with the project they belong to (see nanocoap, which belongs to @kaspar030's sock project), while not desturbing the above mentioned semantics
• It allows for pulling in packages transparently to the user: USEMODULE += lwip_ipv6 can update USEPKG += lwip in Makefile.dep without making lwip all that different from other modules.

[OlegHahm]

So far my understanding of USEPKG was that it is the counterpart to USEMODULE for code that does not resides inside the RIOT repository, i.e. I use USEPKG to download, patch, and build the package.

[kaspar030]

We do need USEPKG because packages are handled differently than modules.

How about adding all USEMODULE entries that have a corresponding folder in /pkg to USEPKG?

[miri64]

So far my understanding of USEPKG was that it is the counterpart to USEMODULE for code that does not resides inside the RIOT repository, i.e. I use USEPKG to download, patch, and build the package.

But then semantics get mixed (and are already) with USEPKG for stuff like lwIP, emb6 and even oonf. Just make a quick grep for oonf_ to see what I mean ;-).

We do need USEPKG because packages are handled differently than modules.

You do it. I didn't do it for lwIP and emb6, since they do have sub-modules and I don't see the point in treating them special in the build process, now that they are checked out per app, per board.

How about adding all USEMODULE entries that have a corresponding folder in /pkg to USEPKG?

So I would add >20 directories for lwip alone?

• pkg/lwip_sixlowpan
• pkg/lwip_ipv6
• pkg/lwip_ipv6_autoconfig
• pkg/lwip_ipv6_mld
• ...

This sounds exhaustive...

[kaspar030]

We do need USEPKG because packages are handled differently than modules.

You do it. I didn't do it for lwIP and emb6, since they do have sub-modules and I don't see the point in treating them special in the build process, now that they are checked out per app, per board.

USEMODULE does:

• add .a to the list of archives to be linked

USEPKG does:

• make -C pkg/ for all, clean, distclean
• include pkg//Makefile.dep
• include pkg//Makefile.include

We might be able to unify that somehow, but that is not the point.

I didn't do it for lwIP and emb6

That's why those packages clutter Makefile.dep unneccessarily.

How about adding all USEMODULE entries that have a corresponding
folder in /pkg to USEPKG?

So I would add >20 directories for |lwip| alone?

I didn't say "create a directory for every (sub-)module a package provides", I meant adding all entries from USEMODULE that happen to have a corresponding package folder to USEPKG.

That would reduce e.g., lwip's main Makefile.dep entries to "if lwip%; usemodule += lwip", and the rest can be done in pkg/lwip/Makefile.dep.

The point here is to remove the distinction from the user perspective, where it is not neccessary. (Unless we want users to realize they're entering the package land.)

[miri64]

USEMODULE does:

• add .a to the list of archives to be linked

USEPKG does:

• make -C pkg/ for all, clean, distclean
• include pkg//Makefile.dep
• include pkg//Makefile.include

We might be able to unify that somehow, but that is not the point.

But why do the make all and make clean need to be called in a different way to how USEMODULE calls its make all and make clean?

That's why those packages clutter Makefile.dep unneccessarily.

Yes, but also because we define our dependencies centrally not module-wise ;-).

The point here is to remove the distinction from the user perspective, where it is not neccessary. (Unless we want users to realize they're entering the package land.)

Huh, all I heard so far is to do exactly that: make modules and packages transparent. For this it shouldn't be necessary for a user to explicitly set the USEPKG macro (which is currently the case because of the semantic fogginess/overlap to USEMODULE of USEPKG).

[kaspar030]

We might be able to unify that somehow, but that is not the point.

But why do the |make all| and |make clean| need to be called in a different way to how |USEMODULE| calls its |make all| and |make clean|?

Because modules are inherently part of our build system.

• all sources of a module reside in one directory
• *[c,cpp,s,S] gets compiled using our cflags, into one .a
• there's no clash in $(BINDIR)/modulename/
• ...

Packages, on the other hand, need downloading, extracting, patching, fiddling, compiling, linking all to the packages liking, usually integrating whatever a package's upstream community deems fitting as their build procedure, into RIOT's.

That could be "unified", but what do you want to achieve, other than either remove duplicate Makefile code (there's not much), or simplifying the user-facing semantics (which an implicit USEPKG for each module un USEMODULE, as proposed, would do)?

That's why those packages clutter Makefile.dep unneccessarily.

Yes, but also because we define our dependencies centrally not module-wise ;-).

Yes, and there's ongoing work to fix that mess. drivers/ started it, now packages can have their own dependency file.

The point here is to remove the distinction from the user perspective,
where it is not neccessary. (Unless we /want/ users to realize they're
entering the package land.)

Huh, all I heard so far is to do exactly that: make modules and packages transparent. For this it shouldn't be necessary for a user to explicitly set the |USEPKG| macro (which is currently the case because of the semantic fogginess/overlap to |USEMODULE| of |USEPKG|).

Good if that's what you heard, as that was my proposal.

[OlegHahm]

Isn't it desirable for a user to distinguish between a module and a package? For me, it is similar as the distinction between a package from official Arch Linux repository and a package from AUR, i.e. with a module I can expect typical RIOT code quality and perfect integration, where a package requires Internet connectivity for first time building, may introduce additional problems, and might integrate a less smoothly.

[miri64]

Isn't it desirable for a user to distinguish between a module and a package?

Yes of course. But currently it isn't clear, what USEPKG is supposed to do and what USEMODULE is supposed to do in this context.

For me, it is similar as the distinction between a package from official Arch Linux repository and a package from AUR, i.e. with a module I can expect typical RIOT code quality and perfect integration, where a package requires Internet connectivity for first time building, may introduce additional problems, and might integrate a less smoothly.

To stick with this comparison: Regardless of its source, in the end you end up with a (local) .pkg.tar.xz file which you then install with pacman (which in my opinion would be comparible with the semantics of USEMODULE). USEPKG on the other hand is comparable to tools like yaourt were the download of a package and the installation are merged into one process. My original concern was, that we do not have a separate step for downloading (like downloading the metadata from AUR and then using makepkg) but directly go for the holistic approach. In my opinion it would be cleaner if the step of downloading would be to visibly separate for the user from the linking step, i.e. have USEPKG just be the downloading step and USEMODULE be the linking step for both modules and pkg (components). Otherwise we risk some very dirty mixing of the two, especially if a pkg has submodules (or should this be called "subpackage" in with your rationalization). This is already apparent in the Makefile.dep.

[kaspar030]

with a module I can expect typical RIOT code quality and perfect integration, where a package requires Internet connectivity for first time building, may introduce additional problems, and might integrate a less smoothly.

IMO, if a package "introduce additional problems, and might integrate a less smoothly", it should be fixed...

But currently it isn't clear, what USEPKG is supposed to do and what USEMODULE is supposed to do in this context.

I'm trying to remove the confusion around USEPKG += jsmn USEMODULE += jsmn. Currently both are required for most packages, which I find confusing, if I'm putting myself into Joe Newuser's perspective. And verbose, from my own perspective.

I propose making USEMODULE += <whatever> implicitly add <whatever> to USEPKG, iff pkg/<whatever> exists.

[miri64]

I propose making USEMODULE += implicitly add to USEPKG, iff pkg/ exists.

:+1: That sounds like a more reasonable direction than the other way around.

[cladmi]

I propose making USEMODULE += implicitly add to USEPKG, iff pkg/ exists.

If that is the case, we should also allow doing USEMODULE += pkg_other_sub_module_inside.

Some packages are implemented by several modules, and I think it is wanted to allow handling the dependencies in their Makefile.dep. If a user is allowed to do only USEMODULE += lwip_ipv6 it would require including all the files in pkg/*/Makefile.dep before setting USEPKG.

Which for me is fine. It only requires protecting all the USEMODULE += in ifneq (,$(filter AAA,$(USEMODULE))) and not expect the package to already be in USEPKG. It would also match what could be required for modules in EXTERNAL_MODULE_DIRS and go more in the direction of, parse everything, handle dependencies, then use the included modules.

---

The other solution would be to still have in the main Makefile.dep things like

ifneq (,$(filter lwip_%,$(USEMODULE)))
  USEMODULE += lwip
endif

[cladmi]

I have a local branch for this and will do a WIP pr for the current state.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you want me to ignore this issue, please mark it with the "State: don't stale" label. Thank you for your contributions.

[miri64]

Still an issue IMHO.

[cladmi]

I had a wip PR in https://github.com/RIOT-OS/RIOT/pull/9307, I was waiting feedback but could work on it again and start splitting things out.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you want me to ignore this issue, please mark it with the "State: don't stale" label. Thank you for your contributions.

[maribu]

@miri64: ping?

[miri64]

I still think this has merrit, but I do not have the time currently to pick up @cladmi's work on this.