west
west copied to clipboard
zephyrproject-rtos
Allow to execute post update script for every project
Allow executing post-update scripts for every project.
manifest:
...
projects:
...
- name: test
post-update: post-update.sh
Or
manifest:
...
projects:
...
- name: test
post-update:
- post_update1.sh
- post_update2.sh
The script should be runned just after the update of the particular project and before next projects being updated. When multiple scripts are provided they should be executed in order of presence.
You can do this with a shell script fairly easily:
for p in $(west list -f {abspath}); do
echo "updating project $p..."
west update $p
echo "doing some post-update steps in $p..."
done
The west list command works in manifest order.
Would you be interested in pre- and post-update hook files, similarly to .git/hooks?
We could potentially add support for .west/hooks/pre-update and .west/hooks/post-update files.
The script is one option but then we have a script to run a tool that runs git.
I want to probably abuse west a bit. Here's what I want to achieve (one of the scenarios, others are even more complex): I have repo A with a structure like this:
|-Dir_a
| |-Dir_aa
| | |-File_1
| | |-File_2
|-Dir_b
| |-Some_other_files
And I have repo B with a structure like this:
|-Dir_aa
| |-File_1
| |-File_2
| |-New_file
What I want to achieve using west is to clone repo A, delete directory Dir_aa in it and clone repo B in this place. In the current state, it won't work because git would complain about files in the original Dir_aa when checking out to needed revision. Allowing for script execution between updating repo A and B would allow for much bigger flexibility - of course, your script still matches this scenario but I wonder if it's a way to go.
Hmm, you're right that isn't a use case I ever wanted to support using west. West isn't against projects nesting within one another (that's a convenient way to emulate git submodules using west for projects that rely on them), but outright conflicts between projects is something I'm not sure we want to support directly. Can you motivate this use case for me compared to, say, creating a branch in repo A with the changes to Dir_aa and fetching that in your manifest?
So let's say repo A is openthread - big external project, part of it is our 802.15.4 driver. In openthread we want to keep the latest stable release of 802.15.4 driver as a copy (no submodules policy). And we need it here as most of the time it's used as a standalone. But we have another project that uses openthread and it should use the latest beta version of 802.15.4 driver. So I want to use west to track the current version of this driver in this project. In both cases, we want to use master of openthread so making a branch requires constant updates in this case.
Another similar use-case - we have some bits of code in one repository that if you paste-replace to the driver it acts as a plugin that extends its functionality and this plugin comes in a few flavors with different behaviors. Default files present in the driver are just simple stubs.
Thanks! So you're saying:
- A == openthread
- Dir_aa == vendor/drivers/802.15.4
- openthread requires a literal copy of every driver in its main tree, so Dir_aa in upstream openthread is a latest stable version
- you want to maintain an openthread downstream where Dir_aa is a beta version
I'm not sure this problem should be solved using west. That feels like bending over backwards to allow openthread to remain a monorepo project, instead of allowing its pluggable dependencies (such as 802.15.4 drivers) be maintained out of tree.
I would say that this monorepo requirement is the real problem here, and it is (predictably) leading to the usual version skew problems that come with such requirements. We designed west exactly to avoid having to deal with things like this in Zephyr.
Why not add a patch to the openthread build system to pick Dir_aa_stable or Dir_aa_beta in your downstream openthread instead? Perhaps using a cmake variable?
just a small note on this.
Although for sure a valuable feature, we must be very careful not to execute any arbitary code as part of west update.
This could become a security risk on peoples machines.
Currently, commands invoked by users can be verified before execution, such as west update and west build, as the code for those commands are already available in the local system.
But users has no way of verifying code executed as post-update, if it is fetched and executed automatically.
Comparing to git, this is also why, users must install git-hooks themselves, and you cannot force a git-hook onto another persons machine.
Comparing to
git, this is also why, users must install git-hooks themselves, and you cannot force a git-hook onto another persons machine.
I agree. This should be set via config options if we're going to support it. Something like this:
west config update.post-update-hook zephyr/scripts/west-post-update.py
But users has no way of verifying code executed as post-update, if it is fetched and executed automatically.
Maybe west could ask for permission to execute a post-update script if one is detected during west update. This would only happen on the very first west update and then again only if the post update scripts are changed after an update.
This would be a mechanism similar to direnv where the user has to call direnv allow before direnv actually source the .envrc file in the current folder.
Comparing to
git, this is also why, users must install git-hooks themselves, and you cannot force a git-hook onto another persons machine.I agree. This should be set via config options if we're going to support it. Something like this:
west config update.post-update-hook zephyr/scripts/west-post-update.py
but that would still allow an west update to change the script automatically, without the user knowing when / why.
And especially when jumping between different forks, as example, users who works both in Zephyr upstream and NCS, could get the post-update script changed without warning.
A small adjustment could be:
west install update.post-update-hook zephyr/scripts/west-post-update.py
and the hook is then copied to .west/hooks folder.
If a hook is changed, that is .west/hooks/west-post-update.py != zephyr/scripts/west-post-update.py
then west update could print a message, such as:
.west/hooks/west-post-update.py is different from zephyr/scripts/west-post-update.py
Consider installing the updated hook.
<print zephyr/scripts/west-post-update.CHANGELOG content>
That way users must still decide manually to update the hook. but they can examine the hook first (just as they can examine a git hook before making it active).
Hopefully it will also make people providing hooks scripts to be a little careful on what to put in such scripts (and when to update), just as people doesn't update git hooks on a daily basis.
This would only happen on the very first west update and then again only if the post update scripts are changed after an update.
except people who switch between upstream Zephyr and a downstream fork (or have any similar workflow). to often get such message, and having to take action.
And having to always press y, makes people lazy and suddenly they just start accepting anything :(
I really like the hook idea mentioned above, but would like if this could be implemented with the same infrastructure as the west extensions. The use case I am currently looking at is where I want to apply a single patch to the zephyr repo (and potentially other repos) - but I do not want to actually clone the whole repo and keep this is sync for a single line patch which is probably only relevant for our use. If there was a post update hook I could create a patcher function in python and register this as a "post_update hook" and check if the current invokation was for the repo I am interested in and apply the patch(es). Keeping the patcher in the manifest repo I could then easily update any needed patches and/or patcher function while updating the west manifest (e.g if the patch no longer cleanly applies at the new revisions)
Keeping the patcher in the manifest repo I could then easily update any needed patches and/or patcher function while updating the west manifest (e.g if the patch no longer cleanly applies at the new revisions)
Sounds like quilt on top of west. Quite a stack-up there.
but would like if this could be implemented with the same infrastructure as the west extensions
So you're saying this would be a west extension command, using the WestCommand API?
Keeping the patcher in the manifest repo I could then easily update any needed patches and/or patcher function while updating the west manifest (e.g if the patch no longer cleanly applies at the new revisions)
Sounds like quilt on top of west. Quite a stack-up there.
Patching is one use case; another could be in our case where we want to generate some python cstructs and enums based on some c header files in our repository - today this needs to be triggered manually before using the python code (we don't have any build stages for the python code atm). If it could be done at the same time as west update we could eliminate a manual step to make sure the python code is in sync with the c code in the repository.
but would like if this could be implemented with the same infrastructure as the west extensions
So you're saying this would be a west extension command, using the WestCommand API?
I was thinking if the extension system allowed pre/post hooks to be registered for all commands or if you were able to override and already registered command such as 'update' with a supercommand which could do the pre/post in a controlled manner and still be able to run the already registered 'update' command at will.
I'll try not to get lost in the weeds of the examples (the python example sounds like something the build system should be handling, but I obviously don't know the details).
I was thinking if the extension system allowed pre/post hooks to be registered for all commands or if you were able to override and already registered command such as 'update' with a supercommand which could do the pre/post in a controlled manner and still be able to run the already registered 'update' command at will.
So it's more like "run this program, specified by executable file or something on PATH, before and after running the command named update", rather than "here is a Python API for doing the same", right?
Did anything come of this issue? I'm looking for a way to automatically apply a patch to one of the repos in the manifest. If not, what's the best practice here? I would hate to maintain a whole repo fork just for a one-line change.
Did anything come of this issue?
While this issue is still open, a number of alternatives have been mentioned above. Does none suit you and why not?
I would hate to maintain a whole repo fork just for a one-line change.
A single branch is enough for a one-line change, does not have to be a "whole repo fork". BTW do you know about west update --rebase?
Similar, more recent discussion:
- https://github.com/zephyrproject-rtos/west/issues/596
I think the issue is that we're using Nordic's NRF SDK, which has its own manifest with Zephyr and other packages. If we need to modify any of the repos included in NRF SDK, we need to block it from being imported in the NRF SDK and then add our own clone. While this doesn't happen often, a bunch of things needs to happen every time we update (pull a new tag, rebase our branch on top, update the reference in the manifest). I think it would be cleaner if we could simply apply a patch where necessary. Then we would only need to update the tag of the NRF SDK manifest.
It's not a huge deal and we're OK with a fork and rebase. But if a more convenient (patch) method became available, we'd probably switch to that.
I think I see the value of some very generic "post update" hook / feature that lets you do anything. On the other hand I would certain not call "clean" a west feature that stacks one version control solution (.patch files) on top of the other, main version control solution (git). "Faster" maybe but not "clean". One of the ways it's not "clean": now you can't really trust the manifest anymore because two different places with the same listed SHA1s now build different versions?
Rebasing a "volatile", downstream branch is very common and it seems every project develops its own tooling to crack this nut, here's one just from the top of my head: https://pypi.org/project/git-pile/. Considering the git community has not yet settled on the "best" way to do this with a single git repo I think it would be very premature for west to make choices here.
Sorry if this is a naive question: why can't you use existing git hooks to automatically (try to) apply a single patch?
Other, related tools:
- https://github.com/git-series/git-series/
- https://git-scm.com/docs/git-range-diff
- https://chromium.googlesource.com/chromiumos/platform/dev-util/+/cebcefdf14dcb9b86/contrib/fromupstream.py (https://lwn.net/Articles/798147/)
- https://thesofproject.github.io/latest/contribute/linux/development_tree.html#development-flow
We could have used git hooks or the suggested tools (thanks for those). I suppose my hope was you'd tell me post-update hooks were on the near-term roadmap, which doesn't seem to be the case.
In any case, point well taken and we'll stick with a fork. Seems like that is the best practice suggested here.