turbolift icon indicating copy to clipboard operation
turbolift copied to clipboard

Add file output to foreach command

Open rnorth opened this issue 1 year ago • 2 comments

Fixes #7

When running turbolift foreach there are several frequent use cases that are ill catered for:

  • reviewing the logs on a per-repo basis - logs are currently in one giant scrollback
  • reviewing just failure or just success logs
  • (very often) gathering the names of the repos where the command succeeded or failed - for example, when running foreach to run tests

To avoid further cluttering the terminal output, this PR:

  • Creates a temp directory each time it's run, which will likely exist until system reboot
  • In that directory creates the following structure, where org/repo/logs.txt is repeated for every repository (mirroring the structure of the work directory):
some-temp-dir
   \ successful
       \ repos.txt        # a list of repos where the command succeeded
       \ org
           \ repo
               \ logs.txt # logs from the specific foreach execution on this repo
           ....
   \ failed
       \ repos.txt        # a list of repos where the command succeeded
       \ org
           \ repo
               \ logs.txt # logs from the specific foreach execution on this repo
           ....

Some notable points:

  • the emitted repos.txt files are suitable for replay back into turbolift using the -r option
  • we don't stop the current logging to terminal output - this is mainly for backwards compatibility

rnorth avatar Jul 23 '24 09:07 rnorth

Great addition, nice one @rnorth ! It will also fix #7 and help with #31

sledigabel avatar Jul 23 '24 10:07 sledigabel

In terms of naming, I was thinking it could be worth adding some sort of timestamp after the repos.txt. The reason is if you use it to replay the process will overwrite the file as it is being read.

Otherwise we need to make sure we tell users to copy the failed/repos.txt before using it in a foreach loop again.

sledigabel avatar Jul 23 '24 15:07 sledigabel

Thanks for the review, I'd like to merge and release today as there's demand to use this internally.

@sledigabel to answer this:

In terms of naming, I was thinking it could be worth adding some sort of timestamp after the repos.txt. The reason is if you use it to replay the process will overwrite the file as it is being read.

The output repos.txt files also go into a temp output directory, so they don't overwrite one another or overwrite the one in the campaign directory. My assumption is that, if a user wants to do something with lists of successful/failed repos, they can copy the contents and do whatever they want.

@Dan7-7-7

Might be an idea to explicitly provide the command to run foreach against only the repos that failed, in case people don't realise that's possible (or even have a --failed flag that does this automatically?) but perhaps not necessary.

I like that - it's a really good idea. My initial thought was "how do we know which the previous run was" but if we were to combine this with @sledigabel's suggestion of timestamped output directories I can see how it could work. I will raise a separate issue, though, as it feels like it would be somewhat complex.

rnorth avatar Aug 05 '24 07:08 rnorth