Migrate off Bintray release distribution

[GreyCat]

As of now, KS is using Bintray as primary distribution mechanism for reference compiler — we publish Linux .deb packages, Windows .msi packages and universal .zip packages there. The only exception is that .jar builds of compiler which go to Maven Central now.

However, it looks like JFrog will close the service pretty soon: on 2021-02-28 Bintray will not accept any new packages, and on 2021-05-01 they will disappear completely.

Any ideas where shall we migrate our releases to?

[brunchboy]

I have found GitHub Actions quite flexible and powerful. I publish a cross platform jar, Windows MSI, and a notarized, code-signed macOS package as release artifacts (both snapshots and final) whenever I push commits to the master branch.

The workflow is here: https://github.com/Deep-Symmetry/beat-link-trigger/blob/master/.github/workflows/uberjar.yml

And the scripts are next to it. I use secrets to store the credentials needed for signing and notarizing. Happy to answer any questions if this seems a promising direction.

Unfortunately I have no idea about hosting Debian packages.

[brunchboy]

And while I don’t yet auto-publish my library projects to Maven central, that looks straightforward, using the same mechanism to manage secrets as I do for the Mac signing and notarizing steps: https://docs.github.com/en/actions/guides/publishing-java-packages-with-maven

I should probably go ahead and add that step, sometimes I forget to run mvn deploy when I ought to. 😆

[KOLANICH]

GitHub Actions generating GitHub Pages will do the trick. I used to be hosting quite some of packages on GitLab (CI + GitLab Pages) this way. GH Actions can host artifacts for pipelines, but this has drawbacks:

• they are compressed into zip archive. It may be undoable, I guess it is Node.JS script does compression before uploading. It'd be nice if someone reverse-engineered that piece of bloatware, so I could create a less bloated impl.
• there is no permalinks, to get a temporary URI one has to use API. And the API is rate-limited.
• there are no links for non-signed-in users. I guess M$ tries to make users into signing up this way. nightly.link fixes that, at cost of trusting that website.
• no FS tree-like structure, so cannot host a deb repo.

So, generating GitHub Pages is mandatory.

[generalmimon]

@KOLANICH:

GH Actions can host artifacts for pipelines, but this has drawbacks

Of course it has, because nobody has ever intended workflow artifacts to be used for compiled executables. See Storing workflow data as artifacts - it's for log files, test results, core dumps and this kind of stuff. Workflow logs and artifacts at GitHub Actions are only kept for a maximum of 90 days, which makes them (among other aspects, as you've mentioned) totally unusable for release binaries.

Much more sensible option is to upload the binaries when creating a GitHub release, as @brunchboy suggested in https://gitter.im/kaitai_struct/Lobby?at=601c2bfc55359c58bf2042e0.

So, generating GitHub Pages is mandatory.

No, it's not. This should be absolutely the last resort.

---

@GreyCat I read the Service End for Bintray, (...) and it seems that they're not closing the JFrog Platform, only some parts of it like Bintray, JCenter, etc. So perhaps we'll be able to just move to another of their services, such as Artifactory which can also serve the purpose? It is claimed to support Debian packages as well.

[KOLANICH]

Much more sensible option is to upload the binaries when creating a GitHub release

I'd like to have an installable binary package publicly available after each commit.

it's a very good way to go and it's very likely that GH pages will ban such usage in future.

I don't think so. I guess they can even introduce a builtin debian repo within packages feature in future. Though IMHO centralysing distribution to GH is not a very good idea. It is clear that M$ is trying to monopolyse even further, and it is only a matter of time when it will exploit their position. The projects should be ready to migrate from M$ by this time. The problem is that there is almost no a sensible replacement to GH / GL. And GH, though proprietary, currently looks more friendly than FOSS GL. At least GH essential features work without JS, and there is no recaptcha, while GL tries hard to eliminate non-js workability and uses recaptcha.

[GreyCat]

Much more sensible option is to upload the binaries when creating a GitHub release, as @brunchboy suggested in https://gitter.im/kaitai_struct/Lobby?at=601c2bfc55359c58bf2042e0.

Yep, but as far as I understand, GH releases lack repo capabilities - essentially, it's just a bunch of files attached to a git tag. It might be ok for .zip and .msi releases, but it won't work .deb / potential .rpm repos.

I guess they can even introduce a builtin debian repo within packages feature in future.

Very likely, but these are not GH pages.

Our options

So, essentially, so far we have several options on the plate:

Self-hosting

Take some kind of http file server (AWS S3 / Azure Storage / etc), put .msi/.zip as file, generate repo metadata around these for .debs, put .deb + metadata as files

• Pros:
  • we'll likely solve the problem for foreseeable future
  • we won't be that dependent on 3rd party repo hosting services
  • publication technology will likely be stable and won't need another touches for prolonged amounts of time
  • we can use our own stable URLs (like https://packages.kaitai.io/) and our own GPG keys for signatures (kind of both pro and con)
• Cons:
  • costs money, which is not bad per se, but means that if somebody (e.g. me) who owns payment disappears, whole project might be jeopardized; can be solved, e.g. by going forward organizing some business entity where there will be more people than me involved legally, but probably that's a whole different topic.
  • somewhat hard to implement, especially if we don't want to have a server permanently running and will rely on S3-style storage.

Joining some existing Debian/RH repo

e.g. deb-multimedia, QGIS etc:

• Pros: will probably work
• Cons:
  • we depend on some 3rd party (which is arguably even less reliable than a company running this as a business)

Build relationships with major OSS hosters

There are servers like https://ftp.heanet.ie/mirrors/ or http://mirror.yandex.ru/ which generally support hosting of OSS project binaries, and it's probably possible to arrange hosting

• Pros: clean and reliable enough; most of these have a huge set of mirrors around the globe
• Cons: still doesn't solve the problem of hosting original http server, as all these are normally operating in mirroring node

Use Ubuntu PPA for .deb

• Pros: Ubuntu will probably continue running this service in relatively long term, should be stable and free.
• Cons:
  • we'll have to run the build itself on Ubuntu infrastructure: will require significant changes to .deb build process.
  • we'll have to use Ubuntu branding / URLs
  • we'll have to use Ubuntu GPG keys
  • not a solution for .rpm (not that we're publishing them at the moment)

Use https://packagecloud.io/

• Pros: more or less the same level of service as bintray
• Cons:
  • switching from one proprietary 3rd party service to another (which might die just as easily)
  • offers some (probably free) hosting to open source software, but subject to discussion via e-mail

Work on integrating with big distros

I.e. get .deb package accepted to Debian / Ubuntu / Mint / etc

• Pros: ideal solution, probably most
• Cons:
  • complicated
  • will likely take years to pass all the gates
  • will likely require significant changes to
  • will likely require tons of preliminary work to get Scala integrated in Debian

[KOLANICH]

Very likely, but these are not GH pages.

Since M$ has already taken steps to encourage people to host packages on GH, I feel like it is unlikely they will take steps against people hosting packages on GH Pages.

BTW: here are the repos already using this lifehack

Use Ubuntu PPA for .deb Cons:

• No TLS, but strongly needed, and it seems Canonical is not going to fix that.

Work on integrating with big distros Cons:

• non-release packages will likely not be accepted

[GreyCat]

@GreyCat I read the Service End for Bintray, (...) and it seems that they're not closing the JFrog Platform, only some parts of it like Bintray, JCenter, etc. So perhaps we'll be able to just move to another of their services, such as Artifactory which can also serve the purpose? It is claimed to support Debian packages as well.

Thanks for pointer! I will check what Artifactory offers, but from a brief glance it looks like they only offer you software to install on certain cloud subscription (i.e. somebody still have to pay for servers running costs) and it's really focusing on private repo scenarios, not public ones.

There is an announce at https://jfrog.com/open-source/ which claims

JFROG PLATFORM FREE FOR OSS JFrog Provides Free Artifact Management, Security and CI/CD Tools for OSS Projects This program will be active very soon. To be notified when you’ll have access to these free tools, Please contact us.

... suggesting that they will have something "active very soon" for OSS, but it's not available yet. And, obviously, given how they've treated OSS once with bintray, I would probably not trust them again.

[GreyCat]

Use Ubuntu PPA for .deb Cons:

• No TLS, but strongly needed, and it seems Canonical is not going to fix that.

I won't see that as a major problem. Anyway, you can always trivially build a SSL frontend over any web server such as ubuntu's PPA.

Work on integrating with big distros Cons:

• non-release packages will likely not be accepted

Yeah, actually, that's a very good point. So, we'll need .deb / .zip / .msi repo for unstable builds from CI anyway.

So, so far I'm inclining towards self-hosting solution, but I would still (1) check the current Artifactory offering and (2) start discussion with https://packagecloud.io/

Another interesting point is figuring out what everyone else who relied on bintray will do. Looks like ~200 of GH projects are also concerned. Specifically interesting discussions (which are not about .jars):

• sbt — https://github.com/sbt/sbt/issues/6294
• webjars — https://github.com/webjars/webjars/issues/1910
• bincrafters — https://github.com/bincrafters/community/issues/1359
• Homebrew — https://github.com/Homebrew/discussions/discussions/691 (they're likely the most impacted)
• souffle — https://github.com/souffle-lang/souffle/issues/1855

[matthewturk]

Yep, but as far as I understand, GH releases lack repo capabilities - essentially, it's just a bunch of files attached to a git tag. It might be ok for .zip and .msi releases, but it won't work .deb / potential .rpm repos.

Are you sure that the GH releases don't include the necessary capabilities? Looking at, for example, the gh project's 1.5.0 release:

https://github.com/cli/cli/releases/tag/v1.5.0

they include a couple different .deb / .rpm release artifacts, in addition.

[generalmimon]

Just for the record, today (at 15:05 UTC) I got the following e-mail from JFrog Team:

Important Updates Affecting Your Bintray Projects [Action Needed]

Hi Petr,

Thank you for supporting JFrog through your use of Bintray. We are and continue to be a community-focused company, and in the spirit of transparency we’re giving you advance notice that as of May 2021, we will be sunsetting Bintray.

Many of the services that Bintray provides are now served by the JFrog Platform and this sunset allows us to focus our development and innovation efforts on building a comprehensive end-to-end solution for hosting and distribution of software artifacts.

To help minimize any potential disruption in service, we have several solutions for you to employ during this transition period to keep your project running smoothly. Below are details on who is affected, how to mitigate any disruption and additional resources for you to contact if you encounter any snags along the way.

Repositories Hosted on Bintray

• Your repositories for module types including Docker, Maven (Java), Npm, NuGet, RPM, Vagrant, and Generic will need to be migrated off of Bintray service on or before May 1st to avoid any disruption in service.

Conan OSS Users

• Conan packages for C/C++ have been migrated to ConanCenter.

Migrate to the Free Subscription of Artifactory

• JFrog has a new free subscription tier that gives you the ability to manage and distribute artifacts with automated security scanning. You can sign up for a free subscription and start migrating your Bintray artifacts. Unlike Bintray, this service also offers Continuous Integration and is suitable for hosting CI artifacts, such as Snapshot builds. For additional details, please read the community announcement.

JFrog wants to thank you again for your continued support. We’re committed to our community and strengthening our offerings. Other questions? Please feel free to contact us on Twitter @JFrog.

Best Regards, JFrog Team

[GreyCat]

@matthewturk

Are you sure that the GH releases don't include the necessary capabilities? Looking at, for example, the gh project's 1.5.0 release:

https://github.com/cli/cli/releases/tag/v1.5.0

they include a couple different .deb / .rpm release artifacts, in addition.

Yep, that's exactly what I've mentioned - they allow to attach arbitrary binary files, and that's it. Real repos offer you a way to subscribe to certain distribution channel and instead of learning about new software from news, visiting a webpage, downloading and installing some package manually, just run something akin to apt-get dist-upgrade from time to time, and know that this ensures that you run latest and greatest.

[lskillen]

Sorry to jump in, but you've also got something like @cloudsmith-io which can offer Bintray equivalence++, and despite being fully managed (i.e. not self-hosted), you can still get some of the self-hosting-like benefits, such as custom domains and custom keys. In that way, you can minimise vendor lock-in by distributing from https://packages.kaitai.io/, but without the hosting headaches. :-) Full disclosure: I work there, but I've felt the pain of vendor displacement!

[GreyCat]

@generalmimon I've got the same invitation too, and I went ahead and experimented trying to create https://kaitai.jfrog.io/

Nothing terribly fascinating so far: you can create "local" repos, you can upload ("deploy") packages manually to it (but this seems to be discouraged and I can't find API to do so from the first glance), but the resulting URL: https://kaitai.jfrog.io/artifactory/debian/ - seems to be private and requires a password.

So, no way to use it for public distribution, or I haven't found a way to make repo public yet.

There also seems to be a repo type = "Distribution" available, which, surprisingly, redirects you to register Bintray account.

[generalmimon]

@GreyCat:

but the resulting URL: https://kaitai.jfrog.io/artifactory/debian/ - seems to be private and requires a password.

So, no way to use it for public distribution, or I haven't found a way to make repo public yet.

It should be possible, see https://jfrog.com/knowledge-base/how-can-i-allow-anonymous-user-access-to-specific-repositories-only/.

you can upload ("deploy") packages manually to it (but this seems to be discouraged and I can't find API to do so from the first glance)

Artifactory REST API > Usage > Example - Deploying an Artifact

[GreyCat]

@lskillen Thanks for the pointer! It indeed looks much more appealing. Let me take a look.

[GreyCat]

@generalmimon

So, no way to use it for public distribution, or I haven't found a way to make repo public yet.

It should be possible, see https://jfrog.com/knowledge-base/how-can-i-allow-anonymous-user-access-to-specific-repositories-only/.

Thanks, looks like I've figured out how to do it. At the very least, https://kaitai.jfrog.io/artifactory/debian/ opens and browseable now. To get it actually to work, this will need creation and signing with GPG key:

Reading package lists... Done
E: The repository 'https://kaitai.jfrog.io/artifactory/debian jessie Release' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

Another non-conventional fact is structure of the repo. Normal Debian repo looks like this:

dists/
dists/jessie/
dists/jessie/main/
dists/jessie/main/binary-all/
dists/jessie/main/binary-all/Packages
dists/jessie/main/binary-all/Packages.bz2
dists/jessie/main/binary-all/Packages.gz
dists/jessie/main/binary-all/Release
dists/jessie/main/Contents-all.gz
dists/jessie/Contents-all.gz
dists/jessie/InRelease
dists/jessie/Release
dists/jessie/Release.gpg
prod/pool/main/k/kaitai-struct-compiler/kaitai-struct-compiler_0.9_all.deb

... and the one that Artifactory generates looks like this:

kaitai-struct-compiler_0.9_all.deb
dists/
dists/jessie/
dists/jessie/main/
dists/jessie/main/binary-all/
dists/jessie/main/binary-all/Packages.bz2
dists/jessie/main/binary-all/by-hash
dists/jessie/main/binary-all/Packages
dists/jessie/main/binary-all/Packages.gz
dists/jessie/Release

[GreyCat]

@lskillen Thanks for suggesting Cloudsmith again!

I wonder if you can help us with which license shall be specified for the repo? Cloudsmith's license choice box seem to lack "GPLv3 or later" (instead offering GPLv3 only):

On top of that, even "GPLv3 or later" would not probably be sufficient for us. We supply a range of components, licensed differently: compiler is GPLv3+ with a few subcomponents licensed differently (MIT/BSD), runtimes are MIT/BSD/Apache, format specifications are CC0, MIT, etc. Which one shall I choose?

[GreyCat]

Overall, except for (hopefully minor) license metadata declaration, I seem to like Cloudsmith's solution :)

Here's what we get ultimately: https://cloudsmith.io/~kaitai/repos/debian/packages/detail/deb/kaitai-struct-compiler/0.9/a=all;d=any-distro%252Fany-version;t=binary/#install

There is one-liner to add the repository:

curl -1sLf 'https://dl.cloudsmith.io/public/kaitai/debian/setup.deb.sh' | sudo -E bash

It defaults to using Cloudsmith-supplied key, and we also can do the same with our own key. And obviously the same can be accomplished manually (e.g. by adding repo to apt sources, adding key, etc).

There is also a CLI in Python supplied to upload packages. We just create a bot least-privileged account and set up our CIs to do so.

[GreyCat]

Proof of concept publishing deb packages to Cloudsmith - looks like everything's peachy ;)

• Changes to repo
• Travis job
• Resulting package uploaded

[lskillen]

Fixed That For You:

We had a look and our automatic sync from SPDX isn't picking up newer licenses added (such as the GPLv3 or Later) because of a change in formatting. So we added it manually in this case. We'll be fixing the automated sync this week, so thanks for pointing it out! Happy to change over the license for you on an existing repository too, just let us know. :-)

The which should I choose is pretty common, but actually the choice isn't as limiting as you think. It's mostly for us to decide and present what the "majority" license is for a repository. Each individual package can still have its own license that differs from the majority license on a repository. So basically, go with the "main" license, and feel free to vary!

[GreyCat]

@lskillen Thanks! Please note that actually SPDX supports expressions: https://spdx.github.io/spdx-spec/appendix-IV-SPDX-license-expressions/ - and quite often licensing might include multiple licenses. If you want to go with SPDX, you'd likely want to plug an SDPX expression parser.

For the "main" license: thanks for explanation! Any chance you guys can tweak the note at repo homepage like this to express the same, i.e. that it's "main" release, and individual packages can be licensed differently? Otherwise it looks somewhat misleading:

Note: Packages in this repository are licensed as GNU General Public License v3.0 or later (dependencies may be licensed differently).

[KOLANICH]

1. Is it impossible to sign the packages and repo metadata with own keys not known to CloudSmith?
2. RSA 2048 is not recommended (not considered insecure by now, but just not recommended), the current recommendations is 3072.

There is one-liner to add the repository:

curl -1sLf 'https://dl.cloudsmith.io/public/kaitai/debian/setup.deb.sh' | sudo -E bash

Piping curl into bash is not recommended. Also the script is too large and redundant. I recommend to do the following:

1. Manually download the public key.
2. Hash it with sha512, remember the hash
3. write a script. In it
   • download the key and verify its hash hardcoded into the script.
   • only after it add the repo into sources.list.d and the key into trusted.gpg.d
   • Pin the repo to that key file using signed-by, as recommended in https://wiki.debian.org/DebianRepository/UseThirdParty . All other repos (including distro ones!) must be also pinned to keys in order to this measure have any sensible benefit for security.
4. Don't publish that script as a file, paste it into a web page and encourage people to read and modify it.

[GreyCat]

Is it impossible to sign the packages and repo metadata with own keys?

I believe that signing packages is done at CI level, and we don't do that currently. Signing repo metadata with custom key is possible do in Cloudsmith.

[GreyCat]

One interesting extra point that Cloudsmith have highlighted is that our .deb packages actually lack "License" line in the metadata. Let me fix that.

[KOLANICH]

Signing repo metadata with custom key is possible do in Cloudsmith.

I hope without disclosing private key to CloudSmith. I mean it can be done on CI too. Kind of:

1. CI fetches the previous version of metadata from remote storage.
2. It verifies own signature on it.
3. It appends a new version metadata, signs it and sends to the storage.

The "public key" (in fact a certificate) comment should clearly state that the corresponding private key is available to CI and that can be compromised and doesn't have much trust. Releases should be signed manually by devs on their own hardware.

I wonder if anything preventing the storage from returning old metadata can be done. It is usually done by setting in repo metadata an interval after which the metadata is invalid. But it is only acceptable to repos reguralily updating the metadata.

[lskillen]

@lskillen Thanks! Please note that actually SPDX supports expressions: https://spdx.github.io/spdx-spec/appendix-IV-SPDX-license-expressions/ - and quite often licensing might include multiple licenses. If you want to go with SPDX, you'd likely want to plug an SDPX expression parser.

You're very welcome! This is actually on our roadmap, although admittedly there are a million things with higher priority right now. 😂 It's likely that we'll move towards deriving the licenses purely from the packages themselves, rather than having to choose the OSS license upfront; but yeah, I'd definitely like SPDX expression support (we have it there for some part of package processing, but it isn't exposed to users yet).

For the "main" license: thanks for explanation! Any chance you guys can tweak the note at repo homepage like this to express the same, i.e. that it's "main" release, and individual packages can be licensed differently? Otherwise it looks somewhat misleading:

This we can definitely do. I'll talk to the team about agreeing on and changing the text here. :-)

Piping curl into bash is not recommended.

The manual steps are provided with each package too; occasionally we see users writing their own installation scripts, yep. The existing setup is really just for "ease of use", but we understand the security implications. Better to write your own and use the bare minimum steps for the ultimate in control. 👍

I hope without disclosing private key to CloudSmith. I mean it can be done on CI too.

It's recommended to let Cloudsmith do the signing because we use it to sign the repository metadata and occasionally peripheral files that you don't upload but I get the reluctance to expose the key. If you're concerned, you could create a signing sub key from your master key, and upload that to Cloudsmith instead. Then you could add a key revocation step to your installation script/process (if you're paranoid), and check if the subkey has been revoked.

It's also worth pointing out that once the key is in Cloudsmith, it isn't possible to easily read it back even if your user account is compromised (or at least, the passphrase). What could they do? Publish new packages or update the key. The paranoid can protect against this with a CI+approval-based approach to publishing, and limit the user that publishes. Then, ensure you protect your user accounts with MFA and limit privileges.

Cloudsmith privileges are based on RBAC, and the org/team model is fairly flexible. Happy to help. :-)

[GreyCat]

@lskillen Thanks for all the updates and chiming in. I'll get to your support as recommended to set up custom domain, set up custom GPG key, and then I would say we should go ahead doing full transition to Cloudsmith repo hosting, future-proof.

@kaitai-io/developers everyone's ok with that or anyone sees any blockers/downsides?

[koczkatamas]

@kaitai-io/developers everyone's ok with that or anyone sees any blockers/downsides?

It's okay on my part as I am pretty inactive nowadays. I assume it has some cons mentioned for other solutions:

• it costs money
• it is pretty new company (13 employees based on LinkedIn, founded in 2016, got bigger funding in '19: £2.1M).
• vendor lock-in (although it's easy to move as we will use custom domain, so it's not that a drawback)

[GreyCat]

@koczkatamas Actually, it doesn't cost us anything: they're offering special deal for OSS projects. Everything else is more or less fair, but given that (1) it's a really easy to migrate (I already did that) and (2) it's not really a vendor lock-in with custom domain, I believe it's looks pretty bright ;)