Embedded JavaScript pipeline PoC

[pda]

trafficstars

Experimental PoC adding a buildkite-agent pipeline eval buildkite.js command using github.com/dop251/goja as a JavaScript (“ECMAScript 5.1(+)”) interpreter, plus github.com/dop251/goja_nodejs for CommonJS require("…") and some other conveniences.

require("buildkite/plugin") demonstrates a native module, implementing the plugin(name, ver, config) JavaScript function as a Go Function.

Other require("buildkite/*") modules are provided by an embedded filesystem, located in a new package resources; it currently contains node_modules/buildkite/hello.js which services require("buildkite/hello") and demonstrates console.log() working.

Other require("…") paths are served from the host filesystem (i.e. the build directory).

There's an example JavaScript pipeline at test/fixtures/pipelines/buildkite.js demonstrating some basic module loading, environment usage, plugin reuse etc:

// Example buildkite.js

// native module implemented in Go
plugin = require("buildkite/plugin");

// loaded from embedded filesystem in buildkite-agent binary
require("buildkite/hello");

const dockerCompose = plugin("docker-compose", "v3.0.0", {
  config: ".buildkite/docker-compose.yml",
  run: "agent",
});

pipeline = {
  env: {
    DRY_RUN: !!process.env.DRY_RUN,
  },
  agents: {
    queue: "agent-runners-linux-amd64",
  },
  steps: [
    {
      name: ":go: go fmt",
      key: "test-go-fmt",
      command: ".buildkite/steps/test-go-fmt.sh",
      plugins: [dockerCompose],
    },
  ],
};

module.exports = pipeline;

$ DRY_RUN=1 ./buildkite-agent pipeline eval test/fixtures/pipelines/buildkite.js
[stderr] 2022-11-14 22:44:37 INFO   Reading pipeline config from "test/fixtures/pipelines/buildkite.js"
[stderr] 2022/11/14 22:44:37 hello embedded FS!

agents:
  queue: agent-runners-linux-amd64
env:
  DRY_RUN: true
steps:
- command: .buildkite/steps/test-go-fmt.sh
  key: test-go-fmt
  name: ':go: go fmt'
  plugins:
  - docker-compose#v3.0.0:
      config: .buildkite/docker-compose.yml
      run: agent

This can be piped into buildkite-agent pipeline upload for now, although eventually it would be integrated.

$ ./buildkite-agent pipeline eval test/fixtures/pipelines/buildkite.js | ./buildkite-agent pipeline upload

[pda]

I've done a bit of refactoring, and some improvements to require("…") loading, and the debug/error messages thereof. You can pass --debug to the buildkite-agent pipeline eval command to get full traces of what files it's trying to load from where.

[davidwheeler123]

I love this idea - I find the yaml quite challenging to work with, and I love the idea of a Turing complete DSL, with factoring and lovely things like that. But why JavaScript vs typescript? With typescript you'd get IDE integration for free, and discoverability would increase 1000%, plus more problems surfaced at compile time instead of runtime

[dabarrell]

Hey @davidwheeler123 – glad you stumbled across this! This was purely a first-pass experiment, and the use of javascript here doesn't mean much – we've also played around with an embedded typescript engine :) This is early days, but certainly a direction we're exploring