Publish smaller packages

[ggetz]

CesiumJS is an extensive library with a lot of functionality that, depending on the app or use case, goes unused. Installing the cesium npm package also syncs third party libraries which may go unused. All features are tied to the monthly Cesium versioning, whether or not anything has changed.

Many users should continue to use the "traditional" CesiumJS releases, but there are also many use cases that would benefit from ingesting smaller, more specific packages.

See this community forum thread for some seeds of this discussion.

• Maintain the traditional combined release as before. There may be some internal refactoring, but from an end user standpoint, the cesium npm package and release would remain as-is
• Maintain all code from within the same repo. We don't want to introduce additional development burden with additional repos or submodules
  • workspaces are built into npm for use cases such as this. It allows dependency management to work similar to before with all modules installed to one top-level node_modules directory
  • All packages would continue to share the same tooling
• Publish individual packages along architectural boundaries. In theory, there should be nothing stopping us from breaking up and publishing code as-is with some additional configuration, however due to some interconnections in the architecture, this may require some refactoring so that we are not "cheating" by using private APIs.
  • Each new package would be able to be published individually
  • Each new package would only require relevant dependencies-- This means dependencies would only be synced if they are strictly needed, rather than the approach we take now where all dependencies are installed regardless
  • Each new package would follow semver, while traditional CesiumJS release could remain with its monthly versioning strategy
• Most major JS projects implement a variation of this (React, Babel, BabylonJS, Angular…)
• While not needed for an initial implementation, eventually, this could allow us take advantage of tooling for dependency graphs and more, allowing us to do things like run only the unit tests relevant to a change
  • lerna is a tool recommended by npm for managing monorepos, also nx for more advanced use cases

A few use cases:

• Widgets
  • CesiumViewer and other Widgets work well for simple use cases that want to get something really basic up quickly. But a more sophisticated app does not need them, and they can actively be a hindrance as they bring in many static assets.
  • Separating widgets would isolate the dependency on knockout, which is generally incompatible with other UI/Component stacks, and itself can be a security issue. Notably, the basic CesiumWidget does not rely on knockout.
• Core/Math
  • Could be used by gltf-pipeline, where we depend on CesiumJS for utilities but don’t use it for rendering (gltf-pipeline is also a dependency back in CesiumJS, which is currently a circular dependency).
• KML, GeoJson/TopoJson, and Google Earth support each rely on specific dependencies that are not used elsewhere in the API

[sanjeetsuhag]

Another benefit this may have is that it would allow us to upgrade our testing tools. We could take advantage of something like Jest, which could run tests in parallel. For stuff that needs to be run in a browser, we could use Playwright or Puppeteer and run tests in all browsers - potentially allowing us to do screenshot testing for things like Sandcastle.

[ggetz]

For the first iteration of this, we'll focus on creating a separate non-widget, engine-only package and a widgets package dependent on the former. This will allow us to set up the repo to support publishing multiple modules while focusing on one set. Separating out Widgets isolates the knockout and nosleep dependencies, as well as minimize and simplify build configuration for those not interested in using the knockout-flavored CesiumJS widgets.

Packages (exact naming TBD)

graph TD;
    cesium/widgets-->cesium/engine
    cesium-->cesium/engine
    cesium-->cesium/widgets
    cesium/engine-->deps[other npm dependencies]
    cesium/widgets-->knockout
    cesium/widgets-->nosleep

New packages should be added as an npm workspace and have their own package.json file. Many projects use a top-level packages folder, but that is not a hard requirement and we should consider what naming makes sense for CesiumJS.

packages
| engine
| | src
| | | Core
| | | DataSources
| | | ...
| | package.json
| | README.md
| | LICENSE.md
| | ...
| widgets
| | src
| | | ...
| | package.json
| | README.md
| | LICENSE.md
| | ...

Some things we'll need to consider for each package:

• Naming - Make sure to run this by the community (and that it doesn't conflict with any other existing packages)
• Entry points - Shipping just the ESM will require a bundler or other build tooling, but will allow for tree shaking and other optimizations, while the upstream "traditional" build can maintain the built Cesium.js.
• Each package should include typescript definitions
• Documentation - I think we want to stick with our original tooling for generating the API documentation, and continue to generate globally. (This is what Babylon does for instance)
• Testing, linting - These can be done either globally for the entire project or centrally for each package, but it should continue to use the same tooling for now. As @sanjeetsuhag mentioned, as we get more sophisticated with our use, we could reconsider tooling and get a bit more targeted. But for now, let's keep the scope small.
• LICENSE.md matching the limited dependencies for each package.
• README.md should include a minimal example and document best practices (for example in Bablyon.js).

[ptrgags]

@ggetz This is exciting to see! Is my understanding correct on these details?

1. Code from Scene or Renderer would end up in cesium/engine, at least for this first iteration
2. Future iterations might further divide cesium/engine into smaller packages, perhaps separating graphics/geospatial math from rendering details

[ggetz]

Code from Scene or Renderer would end up in cesium/engine, at least for this first iteration

Yes. I think it would be cleaner to move source files directly into this directory structure rather than having scripts copy or reference code from Source, but let me know if that's a concern.

Future iterations might further divide cesium/engine into smaller packages, perhaps separating graphics/geospatial math from rendering details

Yes, and this would be split along architectural boundaries such that units of code which require additional dependencies are separated into packages. To do so, we would increment a major version to denote the breaking change of modules moving to a separate package, and potentially cesium/engine would then depend on these new packages (for instance, if we publish cesium/core).

[bampakoa]

@ggetz the initial plan looks great 💯

It may be a bit too early for this but maybe it would make sense to publish the new packages under the @cesium namespace. However, this would introduce breaking changes so I do not know if it is something that could be done atm.

Regarding the testing/linting process, I think it would be better to have tests alongside the code of each package so that it will be easier to find it.

Thanks for the effort 🙏

[mramato]

I agree that switching to a @cesium namespace would be a good call as part of this work.

I'm definitely on-board with the overall intent of the plan, but I wonder if we have actually done any small prototypes (either hacked up Cesium or a small POC complete separate from Cesium) of how this might work. I feel like there could be a lot of potential showstoppers (like JSDoc, TS generation) hiding here that will require some significant work. A small prototype will help expose those problems up front.

nosleep

Do we even need this? Cesium's VR support is ancient and I believe that's the only place this is used. I would definitely encourage us to take a second look at dependencies like this and figure out if the easier solution is to just delete code or remove a feature. How many people are actually using the Cesium VR support that exists today? (and is nosleep even needed for it).

cesium/engine

This alone definitely feels a bit too coarse for me. @kring might have some thoughts here but to me it makes sense that the general rule should be "new dependency, new module" so those geojson/kml/google-earth etc.. modules make a ton of sense for keeping things lean and mean.

@ggetz overall this is a great write-up and looking forward to seeing some proof-of-concepts.

[mramato]

One other thing that comes to mind is public vs private APIs. Right now we have some bad practices where there is a lot of private API usage of the Cesium library. Would moving to modules have the rule that in order for one Cesium module to use the API of another, it has to be public? That seems the only way to dogfood things properly and make a good product. For example, I think even some low level utilities like Check are private, and therefore would need to be promoted to public to be used in "widgets'

[kring]

@kring might have some thoughts here

I don't have too much to add. Since you're planning to stick with one repo (makes sense to me), there's not a lot of cost to extra libraries, so generally going "too fine" is better than "too coarse" IMO. For example, this long-ish list of libraries in cesium-native: https://github.com/CesiumGS/cesium-native#card_file_boxlibraries-overview

(and that's even though CesiumJS does a lot that cesium-native doesn't, like all the GeoJSON / KML / Google Earth stuff Matt mentioned doesn't exist at all in native)

But it also seems reasonable to me to start by breaking out some coarse pieces - even as coarse as engine and widgets, potentially - with the plan to break each of them up further into smaller pieces in the future. I don't think it will be too hard to maintain (temporary) backward compatibility during this evolution.

[ggetz]

Thanks @mramato and @kring!

I agree that switching to a @cesium namespace would be a good call as part of this work.

To publish with the @cesium scope, we would need a new cesium npm organization, which should be free as long as all packages are public.

I wonder if we have actually done any small prototypes (either hacked up Cesium or a small POC complete separate from Cesium) of how this might work. I feel like there could be a lot of potential showstoppers (like JSDoc, TS generation)

@sanjeetsuhag Is currently working on this and we'll update here with results.

nosleep

Do we even need this? Cesium's VR support is ancient and I believe that's the only place this is used.

Yes, this is only used in the existing VR support. We'll consider removing if VR is not getting much usage.

[sanjeetsuhag]

I am currently prototyping on the workspaces branch. I initially tested things on a simple npm package I created, now I'm trying to test things on the cesium repo.

At the moment, I have just broken up the Source folder out into engine and widgets packages, and used the existing tooling to build index.js and index.d.ts for each (although the widgets is slightly broken). For the top level Cesium.js, the gulpfile.cjs is broken, but I expect that will be easy to fix up.

Here's a TODO list of things to test before we can begin to shape this into a PR:

• [x] Hooking up esbuild
• [x] Splitting up Specs
• [x] Get build-docs running
• [x] Refactoring gulpfile.cjs and build.cjs
• [x] Clean up Apps

[javagl]

There hasn't been sooo much feedback on the forum thread. But there's one point that I already mentioned in the forum, and that I wanted to bring up here, in view of the questions about "public vs private APIs" and the "long-ish list of libraries in cesium-native":

Will there be an attempt to align the package structure between CesiumJS and cesium-native?

It is phrased as a question, but from a very high-level perspective, there are good reasons to do that:

• Discoverability and documentation. When people have invested a lot of time in learning a Cesium API, they'd probably appreciate it when they recognize certain structures in a different context. Maybe there could even be documentation pages that contain code snippets that can be toggled between the JavaScript/CesiumJS version or the C++/cesium-native version...?
• When thinking in terms of "building blocks" and "APIs", one could suggestively ask: When it is warranted to have library "X" in cesium-native, what exactly is the reason to not have a library with the same scope in CesiumJS? This refers to granularity levels and blocks like "Math"/"Geometry"/"Geospatial"/"glTF". (Even though, of course, there are things like "CesiumAsync" or "CesiumGltfWriter" that are specific for C++).
• It would allow to add clarity to the dependency graphs (Roughly: The dependency graphs should be "similar"...)

One should not be tooo strict here. And of course, this should not mean to blindly translaterate the API or class structures: The languages and environments are just too different, and in many cases, there will be judgement calls about the exact contents, scope and structure of a library. But for example, when it comes to questions like "Does 'BoundingRegion' belong into 'Geometry' or 'Geospatial'?", one could have a look at cesium-native and see whether it's possible to align both environments on the conceptual level.

[ggetz]

Request for a more minimal lighter-weight distribution requested in https://github.com/CesiumGS/cesium/issues/11323