medley
medley copied to clipboard
Interlisp
Want action automation to build a "nightly" Docker Online image
on days where there has been a push to master (and there hasn't been a manual release of the latest (for either medley OR maiko)
- the trigger for a new release is push to master of medley maiko online notecards
- need something like make. the 'nightly' is to add a lag in case you want to manually run @stumbo @fghalasz 👍or 👎 please.
👍 I think this is doable. I would suggest setting up an automatic release process that runs at a predefined time and checks to see if a push to master has occurred on any of the specified repositories - if not, the process ends. If a push has occurred, the process could then check to see if the latest Docker image matches the current date. If no matching image is found, the process continues and runs the existing Medley Build Process and Docker Build Processes are run to create a new Docker image.
maiko -> medley -> docker -> online
in that order. If any changes (push to master) then build all the rest (in that order). can be started manually or will run automatically nightly. Using dates will misfire if you push / release / push so I think you need to remember the commits that were HEAD when last built.
I created a branch with a new GitHub workflow to further automate the build process.
https://github.com/Interlisp/medley/tree/buildDaily
The workflow is still very much a draft and not ready to be merged into master. However, I wanted to make the branch available for feedback. Also, @fghalasz and I will need to coordinate and make sure we're consistent in naming of releases and setting up workflows so they're reusable.
The new GitHub workflow consists of a series of jobs, each job dependent on its predecessor and each job performing one action in building the complete set of release artifacts. The workflow can either be manually started or is set up as a cron job to run once a day, at 0200 hrs. This time chosen was arbitrary and has no real significance.
Currently only two jobs are fleshed out:
- Determine if the Medley loadups need to be built
- Call the existing workflow to build the Medley loadups.
The others are placeholders to illustrate the overall pipeline.
The job to determine if Medley needs to be built checks to see if the release commit has changed -- to determine if there's been a push since the last time the workflow ran, and also, checks to see if the current release name is old, based off our current naming convention of using medley-yyyymmdd to identify releases. Depending on when the cron job runs, this may be problematic.
There is one additional parameter that needs to be added, whether the predecessor workflow, Maiko, has been rebuilt. How this can be done is illustrated by a couple updates I made to the buildLoadup workflow. I updated it to support being called from the dailyBuild workflow and, also, return a boolean value - true - if the workflow was rebuilt. IN reality, the fact that we called the buildLoadup workflow means it was rebuilt.
Gathering the information to determine if a build needs to occur is done in one job and a second job uses that information and makes the actual call to build the release, if needed. This is a bit clunky but is a limitation of the GitHub Action yml. Likewise, the fact that the shared workflow returns a value is non-obvious, there is nothing in the buildDaily yml that would give you a clue to that occurs. It's a side-effect of the call.
Also, the reference to the buildLoadup file is currently pointing to my fork of Medley - that needs to be changed. And, you'll notice I used a version tag to identify the specific version of the buildLoadup yml file to use.
Remaining work:
- Naming cleanup - I moved some names from snake case to camel case to attempt to make act happy (I use act to run github actions locally and debug). Act currently doesn't do well with reusable workflows and inputs and outputs. I want to undo those changes for consistency.
- medleyPushChecker - I want to revisit this job and see what I can extract out and make common, I think the bulk of the functionality will apply to each component of the Medley release train.
- Write the other jobs to complete the build train.
Re: release tags
In the work I am doing on the docker builds (as well as the automated maiko builds) I used the following scheme to name/tag releases.
Release tag is based on the commit that kicked off the build (or the latest commit in case of scheduled or workflow_dispatch event). The tag consists of: <repoName>-<commitDate>-<commitSHA:8>. The commit date is in UTC. It may or may not be the same as the date that the workflow is happening, which is the current date used to tag releases.
Example: maiko-220121-0f5e5e26
I have an action that creates outputs this tag. It can be found on github at fghalasz/actions/release-tag-action/action.yml.
The action puts out the release tag as an output (under release_tag) as well puts it out to $GITHUB_ENV under the name RELEASE_TAG (useful when doing bash parameter substitution).
The action also puts out an output named repo_name set to the name of the repo (i.e., medley, maiko, etc) since for some odd reason this is not included in the github context even though github.repository_owner is.
I currently call this action using: ....- name: Setup Release Tag ...........id: tag ...........uses: fghalasz/actions/release-tag-action@main
I usually try to parameterize everything in workflows so that they will run either on Interlisp or on any fork (e.g. fghalasz/maiko) with no changes. Unfortunately, one cannot use expressions (i.e., ${{ }}) in uses clauses. So for now (during development on a fork) I have to hardwire in the "call" to this action (i.e., Interlisp or fghalasz or ....). There are ways around this but they are somewhat clumsy and don't seem worth it for this one action.
Hmm looks like tag format in my previous post got a bit mangled (hate this editor!).
Here is the tag scheme without using any brackets: repoName-commitDate-commitSHA:8 as in maiko-220121-0f5e5e26.
@stumbo,
I like your sentry scheme using the cache. We should make it an action that we can use everywhere in the Interlisp workflows.
But, your implementation has one problem: you set the sentry semaphore and the same time you test the sentry semaphore at the beginning of the workflow. But what happens if the workflow fails? If this happens, the semaphore is set saying the build was done - when in fact it failed.
What we need to do is to check the semaphore at the beginning of the workflow and then set it as the last step of the workflow. If the workflow fails somewhere along the line- then the set will not happen.
Doing this is not as straightforward as you might expect because the actions/cache action is set to always both check and set the cache. And as far as I can find, there are no other actions for dealing with the cache.
With lots of experimentation, I did setup two actions: check-sentry-action and set-sentry-action that implement your sentry scheme but with the check and set separated. These actions are currently available in fghalasz/actions repo on github.
The check-sentry-action has an output called release_not_built with values 'true' or 'false'. The set-sentry-action has no inputs/outputs - it just sets the sentry in the cache as you did in your workflow.
(We should create an Interlisp/actions repo for accumulating these common actions across all of our repos and workflows.)
In the fghalasz/maiko repo under .github/workflows is the buildReleaseInclDocker.yml workflow. This workflow uses docker to build the Maiko releases for the various Linux platforms as well as the multiplatform maiko docker image (as well as build the MacOS release). It is implemented using the two sentry actions above so that it will not do the build if it has already been built for the latest commit.
I think this is the build maiko step that you are looking for to call from your daily build Medley workflow.
buildReleaseInclDocker.yml can be called via workflow_dispatch or workflow_run - so that it can be called from your Medley build workflow using a jobs.<job_id>.uses clause. It requires some secrets to be passed from the calling workflow.
buildReleaseInclDocker has an optional input parameter called force (defaults to false) which if true will force the build to happen even if the sentry indicates the build has already been done for the commit.
I am still working on a few kinks out of buildReleaseInclDocker so I haven't moved it over to the Interlisp/maiko repo as yet.
(We should create an Interlisp/actions repo for accumulating these common actions across all of our repos and workflows.)
Isn't Interlisp/.github good for that?
Isn't Interlisp/.github good for that?
That would work. The reference in the uses: clause would then be: Interlisp/.github/actions/name-of-action.
One reason I thought about moving actions, especially shared actions, to a new repo is that we will likely will want to tag and version them at some point in time. Having a repository that just holds actions would simplify that - it would be clear that a version applied to actions and not everything else in the repository.
Actions depend as much on the repository and configuration of files in them as they are occasionally disassociated.
I think we might work better with a project dependency / subproject graph.
Maiko depends on itself and versioning outside (github runners, for example.) Medley depends on Maiko. Online depends on both. Loops depends on Medley, If A depends on B, then when there's a new B you'll eventually want a new A.
I think from the discussion Monday we don't want automated releases just yet. (Until we get more automated testing).
I think that "nightlies" could be automated -- and since everything is going to be rebuilt it's OK to have the dependencies statically encoded in the build order. For tagged releases done manually it's a little more complicated, but not beyond our capability!
