building modules: using python scripts

[pjdobrowolski]

~~Added function for CMake to build modules using python scripts Now using -l switch triggers CMakeLists for module passed after switch. f.e. xtensa-build-zephyr.py -u mtl -k ../keys/mtl_private_key.pem -l smart_amp_test~~

We can build more then one module just by adding more module names.

---------------------> update 08.02.2024 <--------------------------

I've separate building script from xtensa, I also create utils for generic functions used in that spaghetti script. More to come in next commits.

---------------------> update 14.02.2024 <--------------------------

To build modules using pythons were created several tools making building easier. For future deployment there is a tool extracting headers descibed in manifest and creating zip file from them.

modules are build now:

~~python scripts/lmdk/libraries_build.py -l <modules_names> -k <key_directory>~~ python lmdk/scripts/libraries_build.py -l <modules_names> -k <key_directory> To create deployment pack of headers use ~~python scripts/lmdk/header_pack.py~~ python lmdk/scripts/header_pack.py we can build multiple modules by adding them one after another.

---------------------> update 19.04.2024 <-------------------------- Scripts updated with removed env copy part from extensa building python scripts. Scripts moved to lmdk directory to not create multiple locations for the same application.

[pjdobrowolski]

I understand how this can be useful but please don't embed this new, unrelated loop inside xtensa-build-zephyr.py. Isn't a separate build one of the most important LMDK design choices?

You are right. I wasn't sure what community would be say about that.

How about deployment - release build? It is being prepared by python I presume. Modules binaries and exported headers should be gathered in the same place as sof binary. Can it be integrated inside python scripts?

[lgirdwood]

Separate helper script is fine, I assume we may need to share logic from the baseFW script ? if so how ?

[pjdobrowolski]

From my perspective python script for building base_fw needs refactor. It is too long and it has to many functions so it might be a good excuse for clean up, add extra functions etc.

[marc-hb]

Yes, xtensa-build-zephyr.py is ugly and no one likes it. It's ugly because it does all the ugly things that no beautiful person wants to do:

• configuring the Cadence toolchains with their ugly and evil environment variables.
• building the ugly rimage and others
• configuring rimage
• iterating over the list of platforms
• "installing" /lib/firmware/ for Linux. Creating symbolic links
• running west update. That one was a mistake but I lost that fight. Never use -u.
• computing checksums for reproducible builds
• taking out the trash with -p. Don't use that either, delete the build directories instead.

I won't name anyone but I even know people who refuse to use it (and then wonder why their firmware is sligthly different from everyone else's... I digress)

However, all these ugly jobs have one very important thing in common: it's the list of everything that is required to build the "base firmware" - and it's very convenient to have all these in a single place. So there is a tiny bit of beautiful consistency!

So let's draw a line and let's not overload that boat with things required by a new, separate and totally different and independent build.

Divide and conquer - one ugly build at a time! https://mesonbuild.com/Design-rationale.html

A software developer's most important tool is the editor. If you talk to coders about the editors they use, you are usually met with massive enthusiasm and praise. You will hear how Emacs is the greatest thing ever or how vi is so elegant or how Eclipse's integration features make you so much more productive. You can sense the enthusiasm and affection that the people feel towards these programs.

The second most important tool, even more important than the compiler, is the build system.

Those are pretty much universally despised.

The most positive statement on build systems you can usually get (and it might require some coaxing) is something along the lines of well, it's a terrible system, but all other options are even worse.

Immediately contradicting myself a little bit: if there are some specific bits of xtensa-build-zephyr.py that you'd want to re-use then maybe we could split them into some new and shared xtensa-build-library.py or something. But let's not make this giant script even bigger with stuff from a totally separate build system.

Or maybe not because “a little copying is better than a little dependency” (Go proverb)

[pjdobrowolski]

I agree. There are a lot of useful things. I suggest to separate building rimage, deployment to new files, document what are they doing and import them to main script. It would take some time but it be easier to understand and use in future.

[pjdobrowolski]

@marc-hb I removed building from main script.

[pjdobrowolski]

@kv2019i it would be nice, because I would move to loadable up_down_mixer.

[kv2019i]

@pjdobrowolski Can you check the CI fails? There seems to be a lot of failures, e.g. https://github.com/thesofproject/sof/actions/runs/8780400627/job/24090240938?pr=8828

[pjdobrowolski]

Cleaned up.

[kv2019i]

Ready to be merged once require CI is ok. @pjdobrowolski can you check the issue with quickbuild?

[lgirdwood]

Ready to be merged once require CI is ok. @pjdobrowolski can you check the issue with quickbuild?

Hang on, @pjdobrowolski and I have an update thats pending some internal discussion.