hermes

hermes copied to clipboard

Set up automatic software publishing describes how to set up your workflow.

First it creates some branches and a merge request from hermes/curate-result-... into hermes/curate-....

Accepting this merge request runs the pipeline in the branch hermes/curate-.... This branch is of cause not protected and therefore it has no access to protected variables.

But in hermes-ci.yml the HERMES_PUSH_TOKEN (cf. #201) is necessary. Unfortunately the HERMES_PUSH_TOKEN stores a secret and is therefore only available in masked branches.

Making the branch hermes/curate-... protected does not help. This breaks with remote: GitLab: You are not allowed to delete protected branches from this project. Only a project maintainer or owner can delete a protected branch..

This relates to the GitLab example / template.

Yes, I used your gitlab workflow/example/template.

Using persitent branches could allow to setup a gitlab project to allow the HERMES_PUSH_TOKEN to work in these branches.

This would be the same workflow like GitHub Pages Deploy Action which also stores the documentation in a persitent branch which is then used by github pages.

If I remember correctly GitHub Pages Deploy Action uses orphan branches.

Using orphan branches would also prevent merging the hermes branches.

As discussed with @led02 I believe your used workflow has a few drawbacks and could be enhanced. I am talking here from using hermes with gitlab (In general it should be similar in github; but zenodo has a github integration which makes it much easier.).

I see a few possibilities:

1. curating merge request with transient branches (your current workflow)

Committing to the default branch starts the workflow and creates 2 branches hermes/curate-... and hermes/post-.... Further a merge request is created to merge hermes/curate-... into hermes/post-.... The final pipeline runs on merging, publishes the data to zenodo and deletes both branches.

• During the pipelines a few secrets are necessary. Therefore the branches should be protected to allow protected and masked CI variables. But deleting protected branches needs the rights of a maintainer.
• In this way your have to trust the workflow to not delete the default branch.
• Further at least every maintainer can access the secrets. But the zenodo token is a personal access token and should not be shared with other users.

2. curating merge request with persistent branches

This is more or less the same as 1. But the branches (e. g. hermes/curate and hermes/post) are persistent. Maybe this could be configured in the config file. Using orphan branches and store only the necessary metadata would also prevent merging the hermes branches. Using a persistent branch is also done in GitHub Pages Deploy Action.

• The branches can be protected and the pipeline needs only the rights of a developer.
• The default branch is save and can not be changed by the pipeline.
• But still the secretes can be used by every developer of the project. The zenodo token is a personal access token and should not be shared with other users.

3. curating merge request with persistent branches in another project

This would be initially similar to 2. But the persistent branches are living in another project -- let's call it publication project and the other source project.

In the source project the normal workflow with developer, maintainer and merge requests can exists. A push/merge to the default branch would trigger a pipeline in the publication project. The publication project has only one maintainer which is the user with the personal access token from zenodo. To still allow protected branches these should still be persistent. The triggered pipeline in publication project pulls the data from the source project and uses your current workflow with persistent branches (2.) in the publication project. The triggered pipeline in publication project has to ensure no code from the source project is running.

• This allows normal workflow in software development in the source project.
• The secrets (including the personal access token for zenodo) only exists in the publication project. In this way the owner of the personal access token for zenodo has not to share this token and has not to trust the other maintainers and developers of the source project.

4. no curating merge request, simple workflow

Following one typically workflow Create release metadata in a custom script does the curating in a first job prepare_job in the pipeline and in another job release_job the publication is done.

The prepare_job could run on every push and therefore already in a merge request to the default branch the result could be checked by looking at the job output (job artifact or maybe in addition in the job log) -- but would not be inside the repository. Maybe it could also automatically be added as a comment to a merge request.

In this way only a single merge to the default branch is necessary to publish the data.

• Typically workflow.
• Still shares the personal access token for zenodo with all maintainers. In small projects with only one maintainer this would be OK.
• No curating merge request.

5. (no) curating merge request, simple workflow triggering another project

Joining the simple workflow from 4. and using persistent branches in another project (like 3.) allows both advantages.

• Typically workflow.
• No share of the personal access token.
• Curating merge request could be done automatically or manually.

Using deploy2zenodo the workflow/possibility 5. (no) curating merge request, simple workflow triggering another project is demonstrated/used and visualized in deploy_deploy2zenodo_to_zenodo. I'm not sure if hermes could cover this. But perhaps hermes and deploy2zenodo can complement/complete each other.

In https://github.com/hermes-hmc/ci-templates/issues/10 the some problem regarding the secret arised.

your approaches sound good! another option for gitlab might be to work with static environments (https://docs.gitlab.com/ee/ci/environments/index.html). You can create CI/CD-variables that are only available for specific environments, and then you can protect these environments and use deployment approvals. This then at least restricts the access to maintainers and/or admins. But I must admit that I did not yet work with environments in gitlab.

If you then combine this with a repo-specific user account at https://zenodo.org, you can circumvent a misuse of the token. the maintainers of your project (or admins) should anyway be able to create new releases on zenodo.

Thanks!

I also did not yet work with environments in gitlab. But anyway.

Protected environments are only available for Tier: Premium, Ultimate. This may be a drawback.

Otherwise your approach seems to be similar to the described workflow 2. curating merge request with persistent branches. But your approach uses environments instead of branches. Do you know the advantages of environments over branches?

The worksflows 3. curating merge request with persistent branches in another project and 5. (no) curating merge request, simple workflow triggering another project can also be triggered by a maintainer. But with these approaches it is not necessary to share the token. The token is only available in another project and this other project defines how this token is used (cf. triggered workflow). So a maintainer/owner in the project can only do 'evil things' in the zenodo record defined in the other project.

Protected environments are only available for Tier: Premium, Ultimate. This may be a drawback.

oh no! I did not see this. alright this does not work then.

Do you know the advantages of environments over branches?

the only advantage that I see is, that we can use gitlabs built-in deployment approvals when working with environments. but this also needs a premium or ultimate setup. so...

I would then vote for option 2.! Having a separate repository just for publishing to zenodo sounds like too much complexity to me.