Move rill-developer to NPM workspaces

[begelundmuller]

Issue

Currently, the root of rill-developer is a regular NPM project for the Svelte/Node local app with code in build-tools, scripts, src, static, test. Since we have moved to a monorepo, this is both confusing and clashes with other code in the repository. The Go and Java code mainly works anyway (except for minor things like excluding paths from Prettier), but it especially clashes with nested NPM projects like docs and web-cloud (see the notes in the PR).

Specifically, we need this to refactor components into a web-common (or web-core) folder, which both the local and cloud web applications can import from. Also see: Routes and code structure (Notion).

Solution

There are many tools available for dealing with monorepos. Some are cross-language (such as Bazel), some are language specific (such as Turborepo).

In Rill, we'll initially keep things simply by using native build tools – i.e. NPM (for Node), Maven (for Java) and Go. All these tools have very decent built-in monorepo support (through workspaces), and can all co-exist without clashes when configured correctly. In rill-developer, Maven and Go are already configured adequately.

This issue proposes to move our Node code to NPM workspaces (reference). NPM workspaces promote the same kind of project layout as other Typescript monorepo tools (like Nx and Turborepo), so this is a safe first step regardless of whether we'll need a more advanced build system in the future.

Specifically, it proposes the following workspaces:

docs/              # Docusaurus project with new Rill docs
docs-enterprise/   # Docusaurus project with legacy docs
web-cloud/         # SvelteKit project for cloud frontend
web-common/        # Code shared between cloud and local (like components, svelte-query hooks for runtime APIs, etc.)
web-local/         # SvelteKit project for local frontend

Things to do

• [ ] Create folder web-local and move all the files related to the local app into it (incl. package.json, config files, Svelte frontend, Node backend, etc.)
• [ ] Setup the web-common folder – initially, just put a simple "hello world" Svelte component into it.
• [ ] Create a package.json file at the root of the repository that sets up NPM workspaces for web-local, web-cloud, docs and docs-enterprise. Configure the nested package.jsons accordingly. Make sure there's only one package-lock.json at the root.
• [ ] Go through all the top-level config files (like tailwind.config.cjs, etc.) and determine whether or not they belong at the root. Config that stays at root should be general enough to apply to multiple folders in the repo (e.g. to all JS/TS files). (NOTE: You probably need to add a tsconfig.base.json file that nested tsconfig.json files extend).
• [ ] Update any dependent scripts and tests to reflect the new paths
• [ ] Update the CI steps for rill-developer to only run on pushes to relevant files (such as **.js or web-local/**). Travis doesn't have a native paths option, but Github Actions does, so it might be easier to just move to GH Actions (we allow both – with the long term goal of moving everything to GH Actions).

What does success look like?

1. You should be to import components across NPM workspaces. For example, in web-cloud you should be able to run: import { ... } from "@web-common/components/...".
2. You should be able to run scripts in nested workspaces just by adding the -w workspace_name option. For example, npm run dev -w web-local should run the current local app.
3. CI should only run on changes to relevant files/paths
4. The repo root should look tidy and reflect the different components in the monorepo. For inspiration, see the Airbyte repo

Keep in mind that the next step after finishing this issue is to move the explore dashboard components to web-common and import them into web-cloud!

[nishantmonu51]

+cc @himadrisingh : This will have impact on build scripts and we would need your help to update the Makefile.