Easy to use and scriptable cross platform distribution support

[SimonLSchlee]

Point 3 in The Principles of Racket states:

Racket turns extra-linguistic mechanisms into linguistic constructs.

When programmers must resort to extra-linguistic mechanisms to solve a problem, the chosen language has failed them. Even if it is not always obvious how to fix such failures, programming language researchers ought to accept the general idea and try to work on finding the proper linguistic mechanisms. Due to Racket’s uses, the language currently internalizes several resource-management mechanisms that are often found in the underlying operating system. Similarly, this philosophy prohibits the idea of “projects,” as found in other IDEs, because this also externalizes resource management, linking, and other aspects of program creation.

Currently it is possible to create standalone applications by first running raco exe ... and then raco distribute ..., the end result is a platform specific bundle of the application code as byte code, the racket runtime and other resources and libraries.

I propose that we should find a way to make distribution of programs a part of the language (possibly a dsl). So instead of every package having its own ad-hoc method of building release bundles via Makefiles, bash Scripts etc. there is a standardized racket-way of doing that.

• raco distribute should support a platforms parameter: -platforms osx,linux,windows maybe the dsl allows to define all the platform specific parameters you currently give to raco exe and you don't have to call raco exe explicitly anymore?
• new raco command raco publish that has some way of registering user defined hooks/implementations. The idea being that through the hook I can trigger a script, acquire the necessary credentials, possibly via interaction with the user and then proceed to commit and push or upload the release.

Node.js's package.json allows you to define package specific commands that can be triggered, but maybe there is a more rackety way of internalizing these workflow related things.

Another possibility

Maybe the above isn't imagined big enough. Instead of adding more to single packages, we should embrace rackets strengths with multi-collection packages and build another dimension on top of that: meta-packages that describe distributions. Racket and racket-minimal would be two different packages that define what is part of those distributions and there is some command that allows building those distributions from source, or with cached intermediate results and install the distribution or run tests.

This also could make it easier to create use case specific distributions, maybe for a specific racket introductory course with pre configured and customized DrRacket showing the tutorial slides in a split etc.

Are there fundamental flaws in my thinking? Is this a bad idea for some reason?

I think this comment from @jeapostrophe in another issue is relevant to the idea of defining distributions as meta-packages.

[jackfirth]

Relevant related work: Docker's image building engine was recently redesigned to use BuildKit, an engine for deterministic parallel caching builds that uses a low-level makefile-like graph to represent build actions. BuildKit has an explicit notion of "builder frontend languages" like Dockerfiles, Makefiles, and other build tool configuration formats and uses a #lang-like mechanism to convert those kinds of files into BuildKit's representation. Looks like this:

# syntax = docker/dockerfile:1.0
... Dockerfile contents ...

Or for a makefile, like this:

# syntax = someuser/makefile:1.2.3
... a regular Makefile ...

BuildKit uses the # syntax = somedockerimage comment to run a docker container that processes the build file into BuildKit's standard build graph representation, much like the way Racket uses the #lang somemodule line to load a module that parses and expands the rest of a source code file.

So @SimonLSchlee, I think you're definitely onto something valuable here. What if we made a docker image that BuildKit could use somehow to bridge into the world of #langs?

[jackfirth]

Also related is the scripty package, which lets you write shell scripts implemented with Racket that automatically install any packages they need when the script is run. Looks like this:

#!/usr/bin/env racket
#lang scripty
#:dependencies '("base" "somelibrary")
------------------------------------------
#lang racket/base
 
(require somelibrary)

(do-some-stuff)
(do-some-more-stuff)
(do-other-stuff)

[SimonLSchlee]

Related: API for Cross-Platform Configuration My guess is, that this is already used somewhere to create the racket builds. But it would be nice to make it easier to use and see an example of its use.

[SimonLSchlee]

@jackfirth I think there are 2 different cases:

1. packages which are pure racket code and only indirectly use platform specific code (via other packages)

2. packages that directly have platform specific code or libraries

I think docker based tooling would be helpful for 2. because it would allow others to understand and update platform specific packages in a reproduce-able way. For 1. I would prefer cross-compilation that works on the logical racket level. If the current raco exe wraps the racket byte code with the platform specific racket, it should be possible to wrap it with rackets that are specific to a different platform, without running that platform. It would be interesting to know if there are special cases where this is difficult to do. Or if it is simply a matter of downloading the platform-specific minimal rackets and some how fuse them with the byte code.

Independent of using Docker or not, BuildKit definitively looks interesting.

I wonder whether docker based approaches would encourage people to create more packages that for example wrap a game engine written in c or rust, together with bindings written in racket. Maybe there would be a game-engine-lib game-engine-doc game-engine-test and new a game-engine-dev which allows to build the foreign library from source stick the resulting library into game-engine-lib, generate updated docs in game-engine-doc and release new versions of those packages.

[LiberalArtist]

I have a pipe dream that WASM might provide an alternative to distributing lots platform-specific variants of native libraries. I have some (and am trying to put together some others), and it is not delightful to create and maintain them, especially as someone who doesn't work with the incantations for building and linking shared libraries on a regular basis (especially on Windows …).

[jackfirth]

Also relevant is Bazel, the open source version of Google's distributed build tool (internally called "blaze"). That system has a concept of platforms which handles platform-specific dependencies and build actions pretty nicely.

Some time ago I put together a gist with a list of links that might be useful for someone trying to make "multi-platform Racket", in a way that gives individual Racket languages more control over how they behave across different platforms. The gist includes links to lots of articles and specs about Web Assembly, the RISC-V instruction set architecture, Docker images, how Docker handles multi-platform stuff, the Nix package system and OS, how calling conventions are different across different operating systems, and other compiler-related odds and ends.

[mflatt]

@SimonLSchlee Your "another possibilty" is reality. The distribution process is implemented by the distro-build package. The main Racket distribution is a package called "main-distribution". The Northwestern snapshot builds are "Racket plus Tests", because it adds the "main-distribution-test" package. The Utah snapshot site includes "Minimal Racket+GUI+Docs" and "Minimal DrRacket" distributions that are built with smaller sets of packages.

The distro-build package cooperates with the top-level Makefile of the Racket GitHub repo to build from scratch. It takes information about how to build as a #lang distro-build/config program and creates distributions, each for a specified platform and with a set of specific packages. It was specifically designed to support things like course-specific distributions.

[SimonLSchlee]

@mflatt I am curious what parts of Racket prevent it from being fully cross-compiled. For example why it seems to be necessary to run a windows client machine to create a windows build with distro-build. I would especially like to know whether racket-cs has less platform dependencies or may even be close to being able to be fully cross-compiled.

I would be interested to help with creating a minimal-racket distro-build that works completely via cross-compilation, because I think that it would be very attractive for racket to create a "standalone application installer for (gui) applications implemented with racket" for x platforms with as few commands/setup as possible. Without needing to setup build servers and clients, worrying about virtual-machine-images running windows etc.

If you think this cross-compilation goal is difficult (even for a racket-cs only future), that would also be good to know.

For now I will experiment with distro-build and virtual box images. But I am unsure how fast I will have results/time for that.

[mflatt]

The distro-build approach starts from sources, including the C sources for the runtime system, so it needs a C compiler. Currently, distro-build can cross-compile for Windows using MinGW. I don't think there would be any obstacles to cross-compiling Unix variants (not counting Mac OS) with a suitable C cross compiler. Cross-compiling for Mac OS is not doubt possible but more painful.

Meanwhile, as you may know, raco exe can cross-build (https://docs.racket-lang.org/raco/cross-system.html), as least for the traditional Racket implementation. To do that, you need a built version of Minimal Racket for the target platform. The "Tarball" builds at https://download.racket-lang.org/releases/7.4/ and at the Utah snapshot site are meant to support that. It would be nice to have a raco tool that fetches tarballs as needed and otherwise makes the process easier. Also, more work is needed on the bundling side to make Racket CS work; cross-build for Racket CS will require a Chez Scheme compiler binary for each combination of host and target platform. Perhaps, with some work, the Racket bootstrap for Chez Scheme could be used to generate a Chez Scheme cross-compiler on demand for any target platform on any host platform.

Putting those together, it would be possible to generalize distro-build to not build from C sources but instead take advantage of pre-built runtime tarballs (like raco exe does) to populate the new build. That would be the right thing for creating new Racket distribution variants (i.e., things that you want to have a racket executable and libraries), but I'm not sure it's the right thing for creating installers for stand-alone applications that are built with Racket. Building on raco exe and raco distibrute is more likely the right thing for distributing stand-alone applications.