Use GoReleaser for op-node/batcher/proposer/challenger release pipeline

[sebastianst]

GoReleaser is already used for the op-deployer release pipeline.

We don't have a fully automated release pipeline for the main OP Stack components op-node/batcher/proposer/challenger yet. Docker builds happen via CI jobs but drafting the release notes, to only include commits that changed a file in the component's sub directory in question, is still a manual and error prone task. GoReleaser can greatly simplify this step because it can automatically filter to relevant commits.

However, GoReleaser can also fully take over the whole release process, including building binaries and Docker images. We could skip this in a first iteration though, to keep using the current CircleCI Docker build jobs.

I had some limited local success using GoReleaser for op-node with this config:

# yaml-language-server: $schema=https://goreleaser.com/static/schema-pro.json
# vim: set ts=2 sw=2 tw=0 fo=cnqoj

version: 2

project_name: op-node

before:
  hooks:
    # You may remove this if you don't use go modules.
    - go mod tidy

builds:
  - id: main
    main: ./cmd
    binary: {{ .ProjectName }}
    env:
      - CGO_ENABLED=0
    goos:
      - linux
      - windows
      - darwin
    goarch:
      - amd64
      - arm64
    ignore:
      - goos: windows
        goarch: arm64
      - goos: linux
        goarch: arm64
    mod_timestamp: "{{ .CommitTimestamp }}"
    ldflags:
      - -X main.GitCommit={{ .FullCommit }}
      - -X main.GitDate={{ .CommitDate }}
      - -X github.com/ethereum-optimism/optimism/{{ .ProjectName }}/version.Version={{ .Version }}
      - -X github.com/ethereum-optimism/optimism/{{ .ProjectName }}/version.Meta=

archives:
  - formats: tar.gz
    # this name template makes the OS and Arch compatible with the results of `uname`.
    name_template: "{{ .ProjectName }}-{{.Version}}-{{ tolower .Os }}-{{ .Arch }}"
    # use zip for windows archives
    wrap_in_directory: true
    format_overrides:
      - goos: windows
        formats: zip

dockers:
  - id: default
    goos: linux
    goarch: amd64
    dockerfile: Dockerfile.default
    image_templates:
      - "us-docker.pkg.dev/oplabs-tools-artifacts/images/{{ .ProjectName }}:{{ .Tag }}"

changelog:
  paths:
    - {{ .ProjectName }}/
    - go.mod
  filters:
    exclude:
      - "^docs:"
      - "^test:"

git:
  tag_sort: semver

release:
  github:
    owner: ethereum-optimism
    name: optimism
  draft: true
  make_latest: false

monorepo:
  tag_prefix: {{ .ProjectName }}/
  dir: {{ .ProjectName }}

I used it to draft the list of relevant commits for the op-node/v1.13.3 release (I had ascending-by-message ordering enabled, which I removed from above config because I think historical ordering is actually better). Since the tag to release didn't contain the file, I had to move it outside the repo so GoReleaser doesn't complain about a dirty git repo. My invocation was goreleaser release --clean -f ../.goreleaser.yaml.

This config may already work in its templated form for all OP Stack components mentioned above. GoReleaser also supports inclusion of templates, so this could be mostly moved into a template that is then just imported in each component's .goreleaser.yaml file.

However, there are some open issues to sort out.

• We either skip the Docker build, and reuse our existing Docker build jobs, or need to create a Dockerfile that works for the components, possibly also in a templated form.
• I'm not sure if GoReleaser properly selects the correct commit range from current finalized to the previous finalized tag. It includes -rc.N tags in its ordering. We could filter out rc tags when using GoReleaser so it's only used for finalized tags. But this needs to be investigated.

[protolambda]

I think GoReleaser is a great opportunity. But please let's not add a 3rd (4th? 5th?) way of building services before we remove old ways.

Current bloat

Currently we have:

• docker bake file
• makefiles, calling the justfiles, and not passing through all data
• justfiles that then re-invent everything that docker-bake does (git version discovery and build recipes)
• global make recipe to call the docker bake
• other external bespoke ways of invoking the shared service dockerfile while bypassing all of the above

I liked the docker-bake file because it's a native docker format, supports all the configuration and is clear with how variables are sourced. But in practice it's not used well.

I think the jutfile situation is bad, because it duplicated configuration, and makes every service build slightly differently for no reason.

If we add a go-releaser config, let's remove all justfile/makefile/bake references. There should be 1 way to prepare docker images and release builds. Not so many as we have now.

op-deployer config (as-is) is not a good example

Also, the op-deployer makefile/justfile setup still has inconsistencies with git versioning, and a configuration entry-point from the op-chain-ops dir where it used to be located, that is half-broken, like here: https://github.com/ethereum-optimism/optimism/blob/bfb9de2081a8fef8ae998025e1a8ca0510bd5c45/op-chain-ops/justfile#L8

op-deployer also introduces a separate Dockerfile, which is nice in theory, but the Docker build caching and maintainability is only getting worse the more we split it. The ops/docker/op-stack-go/Dockerfile should unify it, and support the proper Go module download and build caching. (does GoReleaser handle docker-layer-caching and Go build caching between different sub-modules?)

GoReleaser is good. Taking previous rushed build systems as example for GoReleaser is not.

And duplicating go-releaser complexity for N services without removing debt here first is even worse. If we do what op-deployer does we end up with N different configs and inconsistent docker base images and build systems etc. to maintain. We need to approach GoReleaser from a monorepo-first perspective.

[sebastianst]

I was primarily looking for a way to automate the generation of release notes, because currently we have to manually go over 100s of commits manually, of which usually 98% are irrelevant to the component that's being released. I saw that op-deployer uses GoReleaser to generate its release notes automatically. It also comes with a whole build+release pipeline, which wasn't clear to me at first. So I just recorded my experience with it here. I'm not pushing to use GoReleaser. I'm actually happy if we just use it for auto-generation of release notes as a first step. We can fix the issues you mentioned later when we find time to streamline the release process.

[geoknee]

Documenting a lighter-weight solution for changelog generation:

git cliff --include-path "op-node/**/*" --include-path "op-service/**/*" --tag-pattern "op-node/*" -- op-node/v1.13.3..HEAD

https://git-cliff.org/ With this template https://github.com/orhun/git-cliff/blob/main/examples/github.toml (only git.filter_unconventional = false).

[geoknee]

Whatever solution we use for generating release notes or changelogs, it would be awesome to commit it to the codebase -- either in the monorepo directly or into op-workbench. The dream would be a one-click solution that makes it easy to ship high quality / accurate changelogs in a consistent format. A lot of our release pipeline is fairly well defined already, so it shouldn't be hard to get the last piece set up in a similarly nice way.

[geoknee]

If we wired up e.g. ./op-workbench release-notes op-batcher to automagically execute:

git cliff --include-path "op-batcher/**/*" --include-path "op-service/**/*" --config-url=https://gist.githubusercontent.com/geoknee/48d0187c591b968fb21b419bc7efb80e/raw/cd402351fd859f9fcd9bdbe97c622f3f25ecb21f/git-cliff.toml --tag-pattern="op-batcher/v1.14.0-rc.2" -- op-batcher/v1.13.2..op-batcher/v1.14.0-rc.2

and even push that up to github via the API. Would be pretty great.