Standardize all building and packaging

[dagda1]

• Reference Issues: (#750)
• Implementation PR: (leave this empty)

Summary

Standardize the building and packaging of all bigtest compiled artefacts.

Basic example

The build tooling should live at a higher level than the bigtest monorepo with as little configuration required as possible apart from existing configuration files such as package.json and tsconfig.json.

An executable file of some description will create the correct compiled artefacts with as few instructions as possible, e.g.

yarn frontside build esm commonjs umd

The executable API should hide any individual bundler information.

Motivation

At the time of writing, all build configuration is set in individual npm build scripts called prepack.

For example, @bigtest/interactor and @bigtest/client both have this prepack script:

"prepack": "tsc --build && tsc --outdir dist/esm --module es2015 && tsc --outdir dist/cjs --module commonjs"

While most other packages have a more bare-bones preack script:

"prepack": "tsc --outdir dist --declaration --sourcemap"

There is too much configuration in each prepack script and no consistency.

There is no standardisation or reuse across existing packages.

Requirements

For this issue/RFC, the requirements are to standardise building only. We can iterate on further requirements after we have one feature complete.

• The build tool should require no additional configuration if possible. The tool should be able to infer a lot from the existing package.json file, tsconfig.json file etc.
• for example, the exports section of the package.json should tell us which types of build are required.
• All package.json files should utilise the new exports field.
• All bigtest packages should be esm packages by default.
• The @frontside/tsconfig base tsconfig file should state that each package starts life as an esm package:

{
  "compilerOptions": {
    "target": "ESNext",
    "module": "ESNext",
    "moduleResolution": "node",
    "strict": true,

• Each package under bigtest should be capable of creating one or all of the following builds:
  1. ESM build
  2. commonjs build
  3. umd

There are 2 separate concerns with regards to building. There is building in the context of running the tests and there is building in the context of pre-publishing. Currently, we do not test the different outputs so yarn build should only perform the default configured build and not produce different targetted outputs.

• Executing yarn build at a monorepo root should build all packages in the monorepo and executing yarn build at the subpackage level should build only that package. The executable running the build command should know what to do by reading the nearest package.json file.
• Publishing packages should execute a prepare step that will build the various outputs such as esm, commonjs, umd. The executable can ascertain what is required by reading the package.json file.

Detailed Design

An executable of some description should be installed into the node_modules/.bin directory that executes the following commands.

The | character below defines a choice of one or a combination of any of the choices.

• yarn frontside build esm | commonjs | umd | static-web | srr-web | all -mode watch | headless | CI
• yarn frontside test unit | integration | e2e | all
• yarn frontside lint ts | js | all
• yarn frontside watch

Commands should be short and snappy and give no clues as to how the commands are implemented or any attached framework. Convention over configuration should be favoured over endless configuration.

Any new package that is added as a direct child of ./packages/ should be included in root level builds of all packages.

Drawbacks

There is a danger of trying to boil the ocean. The solution should concentrate only on bigtest's current needs and not the world at large. Any more significant enhancements are added later in incremental PRs outside of bigtest.

• Scope creep
• Are we making things better?

Alternatives

• find an existing solution, either open source or closed source, such as
  • snowpack
  • ultra runner
  • rushjs
  • nx has a great plugin architecture and might be a good fit

[cowboyd]

Let's add linting to the mix. I feel like it's important that we not need to include eslint and @frontside/eslint-config in each project.

[dagda1]

Let's add linting to the mix. I feel like it's important that we not need to include eslint and @frontside/eslint-config in each project.

agreed and I've updated the text.

[cowboyd]

I think also in the alternatives section, there are some monorepo management solutinos like rushjs that may have some overlap with anything that we would develop in-house

[dagda1]

I think also in the alternatives section, there are some monorepo management solutinos like rushjs that may have some overlap with anything that we would develop in-house

Good idea. Getting something that required 0 development would be nice.

I'll have a scan and update the doc.

[cowboyd]

I'd like to try and capture as much as possible what the developer experience would look like. For example, the current setup, which has now been enhanced with #764 to have clean scripts, watch scripts, etc... should also be part of this document. How would our custom build executable handle all these cases?

Other DX questions we should address:

• Let's say we wanted to add a new package to the /packages workspace. What would be the steps to hook that into the build?
• If I'm creating a new repo that is going to use this tool, how do I set up CI to compile my sources for publication?
• Can we enumerate some of the development scenarios and what workflows this too should support. I'm thinking things like
  • adding feature to @bigtest/server
  • extracting function from @bigtest/effection to @effection/thing
  • adding feature to @bigtest/agent web application

[dagda1]

@cowboyd I would like to see how far we can get with no config and see if we can tell what needs to be done from the package.json and the tsconfig.json and the whole cast of other .config files that are now part of javascript development.

I'd also be wary of adding too many requirements at this stage and instead focus on just one right now which is the building capability first and then we can create see how that went and iterate on the next and create a new issue/RFC, if you are OK with that? If not I will update further.

I have updated the requirements, let me know if this is sufficient

[cowboyd]

@dagda1 Much of this can be speculative, but if we're talking about embarking on a custom build tool, then I want to make sure that we don't end up painting ourselves into any local maxima.

We don't have to implement everything at once, but since this is an RFC level discussion, it pays to rove a bit farther into the future.

[dagda1]

@cowboyd I'ved added a Detailed design section. Please feel free add or change anything.