[Feature-Request]: Support for local checkoutScripts in "--build-only" mode

[sbixl]

For e.g code generators it would be helpful if you could define a local checkoutScript that does not additionally check for changes on a remote repository on every run. If you currently want to embed a code generator in the recipes, the bob project must always be built with "checkout" which means, that in case of changes on the remote, the local SCM is always updated. In most cases, however, you only want to run the code generator. For this reason it would be good if the local checkoutScript is executed when building in "--build-only" mode too.

Is there a possibility to implement this feature in Bob?

[jkloetzke]

Sure. Such a use case with code generators is not too uncommon. So it's definitely a desirable feature.

Theoretically every checkoutScript marked as checkoutDeterministic would be safe to run but that might be too surprising and might unearth recipe bugs. Practically, we should probably add a keyword that tells Bob that a particular checkoutScript is safe to run on --build-only invocations too. Note that there is already the import-SCM that will still update the source even if run with --build-only. So it's nothing completely new.

Possible solutions are:

• checkoutUpdateBuildOnly: the straight forward solution
• checkoutGenerator: is set to true declare that some kind of source code generator is run. If also declared as checkoutDeterministic it will run on --build-only too.
• checkoutNetAccess: if set to false the checkoutScript would have no network access and would run on --build-only because it declared to fetch nothing from the outside.

Any other ideas are highly appreciated. :smile:

[rhubert]

But how to handle recipes were you have a checkoutScript with some parts that should not run in build-only and others that should? E.g. if you get the input from a archive which bob can't extrace. The checkoutScript would be extract && runCodeGen but with build-only you'd want to edit some files and re-run the code gen only..

I think this would require a new script-type. Maybe something like postCheckoutScript running in the source workspace.

[jkloetzke]

I'm not sure if this is a problem that warrants the addition of a new keyword. First of all not the whole joined checkoutScript would be executed but only the recipe/class parts that are marked accordingly. And a checkoutScript must always be prepared to be executed, even if it's unexpected, e.g. due to some tool changes, the user built with -f and so forth. That's why the automatic extraction provided by Bob uses a canary to only re-extract the file if it was updated since the last extraction.

A more fitting example would be a script that first fetches some external resource and then runs the generator. But I don't see any such use case yet.

[sbixl]

I think it would be best if we can add the new keyword checkoutUpdateBuildOnly. I have attached a small example of how a Code-Generator could be inserted, which should also be called in case of a --build-only run too. I have used PWSH, i assume that it also works on a native Linux machine with PWSH support but did not test it. Please run the example with verbosity output because I added some console output for better understanding.

PS: I have used the new keyword checkoutDep which is currently not a part of an official release version.

I have update the example (the old one was not the best), i hope it is more clear now. In the example, the generator does not run direct in a checkoutScript but the generated files provided by a separate package (in which the generator has run) are copied to the appropriate source directory using the checkout script.

example.codegen_2.zip

[mahaase]

I also tested the new feature for our code generators. works fine so far! But I also realized exactly this problem. So I am also interessted into that improvment of the feature 👍

btw: like descibed in the doc, yes, this code generator stuff inside the checkout-step is very confusing! 😉 now I have e.g. 4 directories for the 4 variants, in every of that directories there are the same checked-out idl-sources:

$ bob status
   STATUS M    dev/src/fdt/base/interfaces/21/workspace/fdt.base.interfaces
   STATUS M    dev/src/fdt/base/interfaces/22/workspace/fdt.base.interfaces

for each variant, I have to change the sources in a different directory. 🤔

[jkloetzke]

I guess your code generator is somehow variant dependent. That should be avoided. You could inspect the generator dependencies via bob show --format=diff ... to see what is causing the variants...

[mahaase]

we have 1 recipe, that gets the common idl files. than there are some multipackages to generate

• protobuf files
• header files
• cpp code for a function library
• ...

but for every package, the same idl files will be checked out redundantly.

[jkloetzke]

Then the IDL checkout will somehow be variant dependent from the perspective of Bob. Either consult the audit trail or use bob show to find out why exactly they are different variants....