Release process still has gotchas

[yarikoptic]

woohoo -- 0.17.6 was finally "auto-released" without auto. The last fix needed was #7051 to make secrets available. A few gotchas to be addressed:

• [ ] 0.17.6 tag was not annotated, likely just minted by github! we want to have a proper annotated one. Might even need to just git tag -m "{changelog entries}" "{tag}" and then git push -f {tag} "manually" to overload the one produced by github. Or, better, suggesting scriv (I guess it was the one which triggered it) to do that/similar act.
• [ ] https://github.com/datalad/datalad/releases/tag/0.17.6 text is formatted funny (wrapped)! not sure if our or scriv oddity. if scriv - please report to it.
• [ ] https://raw.githubusercontent.com/datalad/datalad/0.17.6/CHANGELOG.md has problems which should be fixed in current version and reported/addressed in scriv:
  • [ ] <a id='changelog-0.17.6'></a> on top -- shouldn't be in .md!
  • [ ] not even a new line before prior release header # 0.17.5 (Fri Sep 02 2022). might be worth to suggest to use --- divider as we had from auto
  • [ ] probably because of the absent divider -- .rst version lost 0.17.5 section separator completely! https://github.com/datalad/datalad/blob/0.17.6/docs/source/changelog.rst
• [ ] we lost all unicodes (like those bugs etc) which we used to have in header sections. Couldn't they be resurrected?

edit: fixes should be implemented and tested on some test repo first.

[jwodder]

@yarikoptic

• The tag is created directly by the release workflow. I don't see an easy way to extract the changelog entries for use as the tag annotation message.
• The release notes are formatted like that because GitHub treats newlines in release notes (and issue comments and some other Markdown sources) as hard line breaks. The only thing we can do is to modify the snippet generation script to not line-wrap entries.
• Why shouldn't <a id='changelog-0.17.6'></a> be there? scriv seems to do that automatically (with no way to disable it?), and I don't see the harm.

[yarikoptic]

• line wraps -- yes, we better to not line wrap since it also breaks down links so we get funny rendering in pycharm even:
  • may be there would also be away to avoid unnecessary empty lines between entries? is that trailing new line somewhere or scriv added? (then might beg for an issue)
• changelog entries for annotation message -- just all new lines added on top from what it was before running scriv to compose changelog
• I do not see a good reason for html to be hardcoded in markdown, nobody would even see them in rendering and github produces its own ids so it is possible to permalink. So IMHO just makes markdown uglier and overall not really desired.

FWIW the one in datalad-next which is AFAIK generated by scriv: https://raw.githubusercontent.com/datalad/datalad-next/main/CHANGELOG.md

• has no html tags . @mih -- do you strip them manually or they just never came to you? (might be some new scriv feature)
• has nice unicodes in sections. those are straight in section names in https://github.com/datalad/datalad-next/blob/HEAD/changelog.d/scriv.ini#L6 .

@mih --

[yarikoptic]

@mih -- did you use scriv to generate releases on github for -next ? https://github.com/datalad/datalad-next/releases/tag/0.6.0 looks rendered fine despite text wrapping (but smart/human one, not breaking markdown control) in .md (which differs from our observation here). tags are signed so you annotate/sign locally and push and then just convert to release on github right? (that is how I did releases manually as well).

@jwodder I think we might better indeed

• not wrap
• tag locally annotated with content of changelog, push the tag
• trigger github making a release from an annotated tag

i.e. to not rely on scriv+github release mechanism

[jwodder]

@yarikoptic The HTML anchor tags were added in scriv in response to https://github.com/nedbat/scriv/issues/46, which seems like the proper place to comment on your desire for getting rid of them.

[jwodder]

@yarikoptic

trigger github making a release from an annotated tag

What mechanism are you expecting to be used for this? As far as I'm aware, if no release notes are supplied when a GitHub release is created, GitHub will default to displaying the body of the tag annotation or tagged commit message (not sure which) in monospace. Creating a release properly means supplying the release notes in the API request (i.e., ignoring whatever's in the annotated tag), at which point we might as well use scriv.

[jwodder]

@yarikoptic

changelog entries for annotation message -- just all new lines added on top from what it was before running scriv to compose changelog

How exactly should I extract that?

[jwodder]

@yarikoptic Issues with entry separators reported to scriv at: https://github.com/nedbat/scriv/issues/62

[yarikoptic]

What mechanism are you expecting to be used for this

could you please check how auto did that? it seemed to do "the right thing" -- we did get proper annotated tags and releases with markdown and all the niceness. May be it is just a matter of making a release and then force pushing the tag? I did just that now -- made local signed 0.17.6, force pushed to github and then edited the release notes to have proper markdown (no wrapping). Also may be if tag is pushed before minting a release on github , it would create a release without creating a lightweight tag for it? again -- release notes could be edited and proper markdown could be provided.

changelog entries for annotation message -- just all new lines added on top from what it was before running scriv to compose changelog

How exactly should I extract that?

something like this in bash - tune to your liking, probably also removing more header lines since we do not need to repeat the version in the release record etc.

• nlines=$(wc -l CHANGELOG.md)
• update CHANGELOG.md
• newrecord=$(head -n -$nlines CHANGELOG.md | grep -v '<.*>')

[yarikoptic]

re html tags, ok -- let's just keep them for now, not the major hassle.

[jwodder]

@yarikoptic

What mechanism are you expecting to be used for this

could you please check how auto did that?

I'm positive that auto just uses the GitHub API and supplies the release notes in the request body. Note that auto's tag annotations are all of the form "Update version to 0.46.3"; they do not include the new CHANGELOG section.

Is there really a need to include the CHANGELOG in the tag annotation?

[yarikoptic]

I'm positive that auto just uses the GitHub API and supplies the release notes in the request body. Note that auto's tag annotations are all of the form "Update version to 0.46.3"; they do not include the new CHANGELOG section.

it could well be -- how do they get then annotated git tags?

Is there really a need to include the CHANGELOG in the tag annotation?

not really. I just did and considered it a good practice, but does not have to be really. ok to not do it.

I am going to fix up CHANGELOG.md and .rst now -- our CI is failing etc. I will push directly into maint

[yarikoptic]

another gotcha is that if we are to keep html tags, then need to have an empty line after for pandoc to correctly convert them to .rst . Since unlikely to happen, I will just remove tags before converting to .rst. All done in 2de6ee69af1b4d1b31b2afb11f00b1807f963861

[jwodder]

@yarikoptic

it could well be -- how do they get then annotated git tags?

The annotated tags are created in a separate operation from the GitHub releases.

[jwodder]

@yarikoptic So what do you want the tag annotation messages to say?

[yarikoptic]

if not changelog -- keep consistent with auto e.g. Update version to {version}

[jwodder]

@yarikoptic I've successfully tested the requested changes in https://github.com/datalad/temp-release-devel. The latest release includes emoji in the category headers (also in the CHANGELOG), and the lines are not broken. GitHub doesn't seem to display tag annotations in a way that distinguishes them from lightweight tags, but you can still see the tag message by clicking on the ellipsis next to "0.0.2" in https://github.com/datalad/temp-release-devel/tags.

This just leaves the matter of the CHANGELOG section divider.

[yarikoptic]

re tags - cool. Confirmed locally by cloning and

❯ git describe 0.0.2
0.0.2

;) re divider: dropped an idea in https://github.com/nedbat/scriv/issues/62#issuecomment-1256339490 . And we lack new line since we do not have it in our templates/new_fragment.md.j2 so quick fix would be to add empty line after fragment like shown in https://github.com/nedbat/scriv/issues/62#issuecomment-1254873797 .

[jwodder]

@yarikoptic As I understand scriv, new_fragment.md.j2 is only used by scriv create, which we don't use (favoring automatic snippet generation instead).

[yarikoptic]

@jwodder - would you be so kind to go through original checklist of issues and mark the ones which were already dealt with in the datalad/release-action (just ran into nonannotated tag so decided to check if known/addressed)

[jwodder]

@yarikoptic The only remaining issue not fixed in release-action is the section divider issue, which requires changes on scriv's end.

[yarikoptic]

@yarikoptic The only remaining issue not fixed in release-action is the section divider issue, which requires changes on scriv's end.

please file an issue. IIRC also using pandoc there could have provided a solution.

[jwodder]

@yarikoptic I already filed an issue: https://github.com/nedbat/scriv/issues/62.

[mih]

We have just released 0.9.4

This is likely fallout from the switch to maint as default branch.

edit by @yarikoptic: dedicated issue - https://github.com/datalad/datalad/issues/7146

[bpoldrack]

Are you aware of something that still needs doing here, @yarikoptic ?

[yarikoptic]

I think we are ok, thanks!