Use rewatch to build runtime, tests etc.

[cknitt]

Since #7063 and #7068, the infamous ninja.js is gone, and runtime/stdlib and tests are built with the standard ReScript build tooling (rescript.json, rescript wrapper script, bsb/bsb_helper/ninja).

In order to eventually get rid of bsb, bsb_helper and ninja completely, we need to build all these with rewatch instead:

• runtime
• tests/tests
• tests/build_tests/*
• tests/gentype_tests/*

Note:

• Building runtime is a bit special with -make-runtime and the resulting paths in the JS output. When I last tried it with rewatch, I wasn't able to get the same paths in the JS output.
• Even with the standard tooling, I was only able to get it to work as required with all source files in the runtime root. But the goal would be to be able to group them into subfolders (core, belt, js, primitives, ...).

@jfrolich @rolandpeelen could you take a stab at this?

[rolandpeelen]

@jfrolich -- Shall we have a chat about this the coming week / divide & conquer? I might be able to clear up some time to do some smaller tasks

[jfrolich]

Yes, same same! 👍

[cknitt]

After #7593, it can be clearly seen where the legacy build is still used by searching for rescript legacy build.

I had to resort to using the legacy build for the various test repos because with rewatch I would either get panics or missing dependency errors.

[cknitt]

When building the runtime with rewatch on Windows in CI, I get

Can't continue... Found a circular dependency in your code:
Js_dict
 → Js
 → Js_dict

whereas other platforms work fine. 😲

[nojaf]

When building the runtime with rewatch on Windows in CI, I get

Can't continue... Found a circular dependency in your code:
Js_dict
 → Js
 → Js_dict

whereas other platforms work fine. 😲

This was fixed in https://github.com/rescript-lang/rescript/pull/7752/commits/ddb75e3a8296565a935f41de512b9cf917074249

[cristianoc]

When building the runtime with rewatch on Windows in CI, I get

Can't continue... Found a circular dependency in your code:
Js_dict
 → Js
 → Js_dict

whereas other platforms work fine. 😲

This was fixed in ddb75e3

Does this mean we can switch now?

[nojaf]

It's a step in the right direction, but I wouldn't say this PR solves all the problems.

[nojaf]

Okay, another worth talking about is https://github.com/rescript-lang/rescript/pull/7771

The way Rewatch resolves packages is a little different than Bsb.

Rewatch (after my regression PR) looks in three folders to find "dependency" sources:

• <entry package>/node_modules, this in the located cwd.
• <monorepo root>/node_modules, this typically the root repository
• <package>/node_modules, this could be another local package inside the monorepo.

While Bsb will look in the pwd/node_modules and will traverse upward until it finds something ../pwd/node_modules, ../../pwd/node_modules.

This is key reason the legacy builds work in the first place. For example, when we build something like https://github.com/rescript-lang/rescript/blob/master/tests/analysis_tests/tests/rescript.json works is because bsb will just traverse up until it reaches rescript and tries /node_modules/@rescript/react.

We don't seem to do this in Rewatch. So, to build things via Rewatch, we would need to link tests/analysis_tests/tests/rescript.json in the root rescript.json.

Doing that is not so crazy, but the implication is that we need to define things like package-sources and suffix on the root level. So all our project would need to be in tune.

I think this is worth doing, but it does leave us a bit in an "all or nothing" situation I think.

[nojaf]

We discussed this during the latest contributor sync:

• As a last resort, Rewatch should also traverse upwards in the file system to find node_modules, similar to how bsb does it. This should make it straightforward to use Rewatch in the main rescript repository. We'd remove the main rescript.json and let things play out as they used to. Later, we could tighten it up by having a top-level rescript.json and streamlining the usage of package-spec and suffix.
• General advice for monorepo users: have a top-level rescript.json and map out all dependencies. That way all packages share the same package-spec and suffix, avoiding potential non-determinism.
• For v12, we won't change Rewatch to support different extensions per package. We don't know whether bsc can handle this, and we won't investigate it for v12.
• We'll avoid existing crates that resolve node_modules (like oxc_resolver) for now, they're not a perfect fit for locating rescript.json files.

This is how I remember it — please correct me if I missed anything.