Build System Rework #4

[ikskuh]

@mattnite sees reason to rework the build system once again.

I think we're not in a perfect place yet, so that's reasonable.

What are features we need and what are features we want to have?

[taylorh140]

I’m newish. But it took me a bit to figure out what data/files are relevant for the project. For most c projects all the relevant files are placed along side the src for the project. I think I’d rather have a folder init script for my particular micro family. But then again, the current method allows switching between things

[ikskuh]

Idea:

We can ship MicroZig as a monolith, but not as a mono-package.

We did the split originally to reduce package size, and this is still a concern. But with the introduction of lazy dependencies, we can now make a single package the user can pull in, and we can then have an inner resolution logic pulling in all the BSPs lazily.

This will decrease download and cache size, and gives the user a monolithic frontend, which might improve UX a good bunch

Only question is how to distribute the information about BSPs... This might require some "build logic" that glues together the BSPs into a singular data structure...

const BSPs = struct {
  avr: ?type = null,
  rp2: ?type = null,
  stm32: ?type = null,
  …
};

which would be populated lazily with the @import() of the BSPs

[haydenridd]

Really only three pain points with MicroZig's build system in order of most to least painful:

1. Including MicroZig is your project is pretty un-intuitive at the moment (although I understand some of the hackier things we have to do are due to limitations of Zig's build system). Having to include things like:

 .dependencies = .{
        .@"microzig/build" = .{ .path = "../../../build" },
        .@"microzig/bsp/raspberrypi/rp2040" = .{ .path = "../../../bsp/raspberrypi/rp2040" },
    },

In your build.zig.zon isn't very self-explanatory, and in the ideal world I would love to have:

 .dependencies = .{
        .microzig= .{ path, url, whatever},
    },

be the only requirement for including MicroZig in your project, build utilities and all! I think I already have a loose understanding of what prevents us from doing this in Zig's current build system, but I'll spend some time brainstorming.

2. It takes a lot of code searching and head scratching to figure out how everything is stitched together at build time in regards to the chip, hal, board modules as well as startup code + vector table. Lots of tracking down modules importing other modules. It took me a very long time to figure out how const microzig = @import("microzig"); gets you what it does. I think a lot of this can be solved with documentation, but if there's any low hanging fruit to make this more intuitive that would be a welcome change. Note that this one is mostly due to my "firmware engineer" brain not allowing me to trust anything until I know what it's doing under the hood. I'll also think about this one in terms of what could maybe change.

3. Nit, but I can't not read "Board Support Package" when I see bsp/, and none of these are really "BSPs". They're HALs! I think we should rename the bsp/ directory to vendor/, and keep the rest of the organization the same. Both HALs and BSPs can live within this directory (although currently only HALs do).

@ikskuh would your idea basically be accomplishing number 1 in my list? I like the idea of leaning on lazy dependencies to only use what the user wants especially if it allows us to only import a single package to use MicroZig!

[haydenridd]

Oh! Sorry, one more thing that's pretty important to me. MicroZig should support a "native" target (Linux, Windows, etc.). Obviously HAL/device code isn't included in this, it just gives you the bones to make it easy to run certain parts (think testing) or all of your "business logic" code natively. I have a complicated build setup on my current projects in C/C++ that lets me do this and it's a dream once you get it working. You basically always have a dummy, simulation version of your firmware that runs on your computer.

I'll take on brainstorming how to do this, since I currently haven't even done this vanilla in Zig's build system. I think Zig's build system and language is very well suited for this purpose though. Lazy evaluation + easy comptime switching based on architecture should make this much more ergonomic than C/C++ + CMake.

[ikskuh]

@ikskuh would your idea basically be accomplishing number 1 in my list? I like the idea of leaning on lazy dependencies to only use what the user wants especially if it allows us to only import a single package to use MicroZig!

Yes, that would be addresses by the "entrypoint/mono-package" style package. No idea how we could make that yet, but it should be doable for sure. We just need to keep the vendor packages in sync with the root build script, but that's totally fine i guess

Nit, but I can't not read "Board Support Package" when I see bsp/, and none of these are really "BSPs". They're HALs! I think we should rename the bsp/ directory to vendor/, and keep the rest of the organization the same. Both HALs and BSPs can live within this directory (although currently only HALs do).

I'm in for that, i don't like "bsp" either. We could also use port or arch or something like that

MicroZig should support a "native" target (Linux, Windows, etc.).

I agree, it's super useful for testing and simulation, which makes it easier to run improved tests

[haydenridd]

I'm in for that, i don't like "bsp" either. We could also use port or arch or something like that

Yep port seems perfect.

I think once I get RP2350 in a somewhat working order I may take a break from that to noodle on the build system. It makes my head hurt every time but there should be a way to get what we're after and ditch the need to patch all of the relative imports into URLs... Well, hopefully :)

[taylorh140]

I was thinking that using git submodules might help with the zig zon archives, and allow keeping somewhat the same monorepo. Sadly they are a pain.

[ikskuh]

I was thinking that using git submodules might help with the zig zon archives, and allow keeping somewhat the same monorepo. Sadly they are a pain.

Nah, not really, as you then have to double-maintain all dependencies, while .path is a maximum in MicroZig Development Experience for now

[taylorh140]

I was thinking about the people who are working off of main. They have an additional steps downloading extracting the repo and placing somewhere convenient. But basically it’s because “.url” doesn’t support a relative path on zon. I feel that this could be potentially fixed by using CI to build the individual root packages (artifacts). It would help with release consistency, cause individual revs could be used.

[mattnite]

I'm liking what I'm seeing, to summarize some action items:

• Rename bsp to port
• Single entry point into microzig
• Add a native target

For the single entry point, I think that's what we need, we just need to figure out a way to make it work with lazy dependencies, so we need some sort of runtime registration. Now another question for y'all, up until now I've been big on users being able to set up their own hardware support if they didn't want to deal with upstreaming it. But I'm starting to question that, why put work into that part of the design for people who aren't going to give back to the project? Like maybe we still expose that ability, only if it's trivial to do so. What do you think?

For the native target, I know @vesim987 wants it so that he can leverage the GPIO interface on a raspberry pi. Is what you're referring to @haydenridd more about software simulation where I imagine you provide some fake hardware? (I suppose vesim could wire the gpio interface to his OS).

[haydenridd]

For the native target, I know @vesim987 wants it so that he can leverage the GPIO interface on a raspberry pi. Is what you're referring to @haydenridd more about software simulation where I imagine you provide some fake hardware? (I suppose vesim could wire the gpio interface to his OS).

Yup exactly, for the dev phase where you're still waiting for boards to come back but just want to test your business logic. Essentially just give you an easy way to compile a native executable. Obviously it's up to the user to correctly swap out their hal for a dummy simulation maybe using a cfg.is_native compile time constant MicroZig provides. There is also maybe an argument to be made that this is up to the user to do, as there's nothing that keeps you from swapping out add_firmware() for the standard Zig build commands based on an input target (native vs. an MCU) in your build.zig. I can noodle around with some stuff...

[ikskuh]

I would use cfg.is_simulation instead of cfg.is_native, because native might still be a suitable target (see: raspberry pi)

For the single entry point, I think that's what we need, we just need to figure out a way to make it work with lazy dependencies, so we need some sort of runtime registration.

For the GitHub record, I started drafting a new build interface here: https://github.com/ZigEmbeddedGroup/microzig-build-api-draft

My current focus is on composing via b.dependency() and a custom MicroZig instance

[tact1m4n3]

The build system is centered almost completely around the build/build.zig file. This I think cuts off a lot of flexibility in terms of what it is capable of.

This, for instance, is very hard to achieve.

... in the ideal world I would love to have:

 .dependencies = .{
        .microzig= .{ path, url, whatever},
    },

I suppose a better way to write the build system would be to have each part (chip, board, hal, even cpu although including it in core is fine I guess) have their build.zig file do the actual build of the module. Then we could have a master module (it can be the root module of the repo) that would create the hierarchy made currently by the core module importing all the necessary parts, including the user's app (the root module's build.zig should contain the current add_firmware etc. functions from build/build.zig). In the root module's build.zig.zon we could have all the chips, boards, cpus (maybe) as lazy deps so as to be able to add them as imports. Another benefit of this is that modules can depend upon each other (the hal can depend on chip, chip on cpu, board on hal).

An issue that I can spot in this way of doing things is that there could be some code repetition in each build.zig (in all chips for instance as we compile the registers). But I don't think this is a significant issue.

What do you think? I'll be glad to implement this, but I thought it would be better to ask before, so as to not work in vane.

[tact1m4n3]

I also think this would improve a little the readability of the repo. It took me quite some time to learn how things are glued together.

[ikskuh]

What do you think? I'll be glad to implement this, but I thought it would be better to ask before, so as to not work in vane.

If you figure out a way how that would work, sure!

Problem is that you want to access a microzig package from all modules (chip, cpu, hal, board, core), so you have to create the packages per application and not once in the build.zig file of it's "owning" project.

[tact1m4n3]

Problem is that you want to access a microzig package from all modules (chip, cpu, hal, board, core), so you have to create the packages per application and not once in the build.zig file of it's "owning" project.

This is something that puzzles me because I don't really see why those have to access the whole microzig package, but maybe I miss something. Like the hal package can have a dependency on the specific chip and cpu instead. We could join everything (hal, chip, cpu, util etc.) after the fact, only in the api that we expose to the user.

Also, I think the way that stuff currently works allows for ports that live outside the microzig monorepo to be used. Is this a kind behaviour that should be kept? I think it's a nice feature.

[ikskuh]

This is something that puzzles me because I don't really see why those have to access the whole microzig package, but maybe I miss something. Like the hal package can have a dependency on the specific chip and cpu instead. We could join everything (hal, chip, cpu, util etc.) after the fact, only in the api that we expose to the user.

The reason is that a HAL can be used for many chip packages, and a chip package can be used for many CPU packages.

Or considering the RP2350, you can use the same HAL and chip package for two distinct CPUs and it will still work.

[tact1m4n3]

I'll get to work then :)) I have a couple of ideas for it. I'll post updates in this issue.

EDIT: Some things that I mentioned in the first message, on second thought, I don't think I am gonna do. I'll try to focus on adding lazy dependencies support without messing with the project organization.

[MatthiasPortzel]

Should this be closed, with #259 being merged?