Suggestion to migrate CI tests to GitHub

[ikappaki]

Hi,

I believe there is some merit migrating the CI tests to a GitHub workflow, is there any appetite for it?

1. Easy access via the Actions menu in GitHub.
2. Currently macos tests are running on GitHub while the rest run on CircleCi, we can get test results under a single badge.
3. GitHub appears to give unlimited credit for running jobs, thus we can extend the test coverage to include all supported versions across all architectures.

I have created a simple demo workflow to showcase the outcome https://github.com/ikappaki/cider/blob/a4de8824d8e3cdbc8756660fab78f883eea60bfd/.github/workflows/test.yml

It is configured to test what I believe are CIDER supported Emacs versions across macos, ubuntu and windows. There is also Emacs master setup only for ubuntu.

Thanks

[bbatsov]

I'm totally fine with the idea of moving to GHA. (mostly because I don't like mixing CI tools across my projects and in general I've been slowly moving in the direction of GHA)

[vemv]

I'd strongly oppose this for the simple reason that I've spent significant effort creating remarkably comprehensive setups in various clojure-emacs projects.

Having one's CircleCI expertise go to waste is no fun. Especially as I also use that expertise in other projects that also benefit the Clojure community e.g. Eastwood and Cloverage.

Significantly, we now release via CI (instead of a manual lein deploy) in a few of our projects.

And lastly, CircleCI can be seen as a premium product crafted by fellow Clojure developers.

Contrariwise GHA is comparatively young, and very importantly is used by too many non-paying developers, so it will have more downtime, more noisy neighbors, and so on.

Going point by point:

Easy access via the Actions menu in GitHub.

Doesn't seem an important point - people are used to scan for badges in the readme. I'm also not enthusiastic about giving Github a monopolistic advantage (if they were fair, they'd have a generic CI button).

Currently macos tests are running on GitHub while the rest run on CircleCi, we can get test results under a single badge.

Also not an important point. Large projects can have ~10 badges or so. We have fewer.

GitHub appears to give unlimited credit for running jobs, thus we can extend the test coverage to include all supported versions across all architectures.

Right now we're not running into problems related to comprehensiveness or credits.

---

Other than that, thank you for your continued contributions in this area 🙌

[vemv]

A tdlr would be to look at the big picture - what do we want for clojure-emacs (which most likely benefit from homogeneity), what do we have now, who is actively maintaining their CIs?

From that point of view, I wouldn't want to introduce problems while our current situation is non-problematic. Churn is expensive, and our time is limited. I'd love to spend the little "OSS time" I have on actual feature development.

[ikappaki]

A tdlr would be to look at the big picture - what do we want for clojure-emacs (which most likely benefit from homogeneity), what do we have now, who is actively maintaining their CIs?

From that point of view, I wouldn't want to introduce problems while our current situation is non-problematic. Churn is expensive, and our time is limited. I'd love to spend the little "OSS time" I have on actual feature development.

HI @vemv, no worries, this was only a suggestion anyway that I brought up for discussion. I thought of it as an improvement over the current CIDER CircleCI setup -- not that CCI is problematic, but GHA feels better IMHO. I've created a GHA setup with the last three version (26.3, 27.2 and 28.1) running all tests/linters/byte compilers accross macos, ubuntu, win and a snapshot run for Ubuntu. I will keep the branch up (that I was just about to PR) for a while in case it comes up at a later while considering the bigger picture or there is a change of sentiment: https://github.com/ikappaki/cider/tree/tech/github-ci-all

Thanks

[bbatsov]

Generally we can have consistent setup across all of our projects on every CI. As I'm still on my vacation I'm not in the mood for writing essays on the topic, but I'd like us to focus here on more technical merits and less philosophical ones - e.g. is there something in our current setup that can't be done with CircleCI, are there known problems with GHA and so on.

As mentioned in other tickets - I don't have anything against CircleCI, but I'm not particularly attached to it either. I've been observing a mass exodus from CircleCI to GHA in recent years and I'm wondering if we won't be fighting an uphill battle here. Let's keep the discussion going.

[ikappaki]

Generally we can have consistent setup across all of our projects on every CI. As I'm still on my vacation I'm not in the mood for writing essays on the topic, but I'd like us to focus here on more technical merits and less philosophical ones - e.g. is there something in our current setup that can't be done with CircleCI, are there known problems with GHA and so on.

As mentioned in other tickets - I don't have anything against CircleCI, but I'm not particularly attached to it either. I've been observing a mass exodus from CircleCI to GHA in recent years and I'm wondering if we won't be fighting an uphill battle here. Let's keep the discussion going.

Here are some of my personal observations. To me the holy grail is to have all tests run across all supported Emacs versions and architectures:

1. On CircleCI, we use the chocolatey package manager to install Emacs on Windows, it only supports one version the latest, we can't test on earlier versions. All other install methods on CircleCI known and considered (msys64, scoop) similarly only support the latest Emacs release. Thus we are limited testing against one Emacs version only. Moreover, chocolatey is still on Emacs 27.2.
2. If I remember well, some time ago, @bbatsov mentioned that we were running out of free credits on CircleCI because of the number of jobs submitted, and some were temporarily halted? (but please correct me if I remember wrong).
3. I also endorse @vemv 's points in defence of CCI:

And lastly, CircleCI can be seen as a premium product crafted by fellow Clojure developers.

Contrariwise GHA is comparatively young, and very importantly is used by too many non-paying developers, so it will have more downtime, more noisy neighbors, and so on.

Thanks

[bbatsov]

If I remember well, some time ago, @bbatsov mentioned that we were running out of free credits on CircleCI because of the number of jobs submitted, and some were temporarily halted? (but please correct me if I remember wrong).

Yeah, that's true but I think they gave us more build credits in the end, so it's no longer an issue.

[ikappaki]

Here is what appears to be another limitation/restriction with CCI. I can't see the CircleCI results on the PR, and I assume this is the same for the reviewers? i..e the reviewer does not know whether the changes is passing CI tests? It only shows the results of the GitHub Actions:

[vemv]

I think CCI does not run if you already have a forked project running in your personal CCI account

i.e. CCI won't build a project twice (one for your fork, another for upstream).

Generally this isn't an issue as most people won't set up a CCI project for their forks.

---

Edit: GHA is not too different - it requires explicit build activation for forks, because it also doesn't want to run things too much

[vemv]

I'd like us to focus here on more technical merits and less philosophical ones

We could even start by wondering if there's a problem at all to begin with. If not we can compare all day long but not be solving a problem.

The only practical problem I perceive is that checks are not enforced for merging, so CI has a limited role in preventing mistakes. At the same time, that topic has been discussed time and again. Personally I'm quite fine with things as they are.

[ikappaki]

I think CCI does not run if you already have a forked project running in your personal CCI account

i.e. CCI won't build a project twice (one for your fork, another for upstream).

Generally this isn't an issue as most people won't set up a CCI project for their forks.

Edit: GHA is not too different - it requires explicit build activation for forks, because it also doesn't want to run things too much

Hi @vemv, just for me to understand the behavior, if I haven't had a CCI account (or I didn't enable it somehow for a particular project), the CIDER PR would have run the tests on CCI and thus the results would be visible to reviewer? Is this what's happening on the other open PR (number 3201) and I as such I can see the CCI results on the PR? I was puzzled by this ...

Thanks

[vemv]

Yes, in order for builds to show up you have to disable your project fork in your CCI account

(having a CCI account of course doesn't disable builds per se)

[vemv]

We're probably happy with the current mixture of CIs, will tentatively close this issue.

Cheers - V