A new strategy for templates and versioning

[danielledeleo]

Related issue: #167

Background

What is a Template?

A Template can be any one of these things

• Runnable templates for each language
• Atmo project template
• Docker Compose template (for Compute)
• Kubernetes template (for Compute)

These templates are currently kept in one of two places: in the suborbital/subo repository in the templates directory, or in a separate repository with the same structure (e.g. suborbital/runnable-templates). By default Subo grabs its templates from the main branch of the Subo repository. The repository and/or branch can be overridden:

subo create runnable my-runnable \
    --repo jagger27/runnable-templates \
    --branch somebranch \
    --lang tinygo

Why do we want customizable templates?

If you want to include extra libraries in your Runnables you need a custom template. This is important for Compute clients, who may want to provide something more specific to their use case than the default Reactr func run(input []byte) ([]byte, error) Runnable signature. For example, in Rocketship we use a customized template that presents the user (i.e. the person writing Runnables) with a slightly different signature:

import { JSON } from "assemblyscript-json";
import { logInfo } from "@rocketshipinc/rocketship"

export function run(input: JSON.Obj): ArrayBuffer {
	let payload = input.toString()
    // ...
}

Compared to default AssemblyScript template has this signature:

import { logInfo } from "@suborbital/suborbital"

export function run(input: ArrayBuffer): ArrayBuffer {
	let inStr = String.UTF8.decode(input)
    // ...
}

Problems

Subo keeps a local cache of templates and does not update them by default.

This has caused a bunch of issues in the past and has the potential to break things very badly in the future. By default, for example, a fresh installation of subo downloads its templates from suborbital/subo on the main branch, which is not guaranteed to be stable or compatible with the builders that subo spawns. Most of the time it does work fine because Runnable templates aren't updated very often and the Docker image tags themselves are parameterized in the templates.

Because templates are not updated by default, it is possible to get stuck with old templates because we cache them aggressively.

The --update-templates flag exists to force an update for this reason.

There isn't a nice way to get old templates

subo always fetches from main. It would be good to be able to fetch older templates for compatibility reasons.

This is less of a problem for local Runnable development, but it can be a problem in SCN where the code snippet is injected into a fresh template every time. Runnables are pinned to a specific Reactr version, but the templates are not. It should always be possible to build a Runnable with the template it was first written for, and optionally upgrade the template if that is desired.

The runnable-templates idea only applies to Runnable templates.

As mentioned above, subo uses its templates directory for more than just Runnables. Atmo projects and Docker/K8s definition files are also generated by Subo. I suggest using renaming that repo to subo-templates so that it is named better.

Runnable templates are pretty hard to write

TODO.

Tasks

Subo CLI tasks:

• [ ] Subo should only ever fetch templates that match its own release tag unless told otherwise.
• [ ] Subo should detect out of date templates.
• [ ] Subo should be able to fetch templates from other git hosts

Tasks related to moving templates into suborbital/subo-templates

• [ ] Templates should live outside of the Subo repository so that they can be forked more easily.
• [ ] This repo could have a GitHub Action that automatically attempts to pull updated base templates in for forked templates so that they don't get stale.

[rnpridgeon]

Would it make sense to not rely on the zip at all but instead to introduce the concept of a template "source" .

If we have a dedicated templates directory, which we judiciously tag, we could just pull the template as needed.

In my head it looks something like this:

1. I am ready to create a new function
2. I specify a lang with lang
3. I specify a template source with --template https://github.com/not_suborbital/runnable-templates/blob/v0.2.0/templates/rust/src/lib.rs.tmpl

We could consolidate 2 and 3 if we come up with a well-defined template structure and we could optimize suborbital template repositories .

i.e. --template v0.2.0 will be translated to --template https://github.com/suborbital/runnable-templates/blob/v0.2.0/templates/rust/src/lib.rs.tmpl

Just food for thought, either way excellent report.

[danielledeleo]

@rnpridgeon I like this idea, but I think I would still rather pin the versions to git tags.

The main problem I see is that templates are more than a single file. Maybe it could point to a manifest file in that subdirectory?

Something like...

template.yaml

name: tinygo-db
language: tinygo
description: A sample TinyGo Runnable with database capabilities
compatible:
  reactr: >0.15.0

and then the actual template files could be stored in a standard folder structure adjacent to the manifest file.

The major benefit I see for something like this is that we could have multiple templates per language. It would also be neat to have "base templates" that could be imported and composed together, but that's probably too wild.

[rnpridgeon]

Yep you get the idea, we come up with a well-defined structure for template metadata and use that as a means to reference a specific template resource itself.

I haven't put a terrible amount of thought into exactly what that looks like but the idea is to keep metadata and data separate while maintaining a stable relationship between the two.

[arbourd]

I think it would be helpful to bake in the release templates, into the release binary. We can still give people the ability to specify a repo/branch or zip somewhere, but most people aren't going to need that. This way for 99% of users, we can assure that v0.4.2 of subo is using templates from v0.4.2.