cue/load: support alternative loading mechanisms

[myitcv]

Is your feature request related to a problem? Please describe.

Currently, per https://cuelang.org/docs/concepts/packages/:

Within a module, all .cue files with the same package name are part of the same package. A package is evaluated within the context of a certain directory. Within this context, only the files belonging to that package in that directory and its ancestor directories within the module are combined. We call that an instance of a package.

We refer to this as the "k8s loading mode". Per #2000, it is not currently possible to turn off this behaviour.

Nor is it the case that this loading mode covers all situations. Two examples that we will refer to as the "Vector loading mode":

• The Vector project distributes files in a single package across multiple subdirectories, reifying these into a single package value via find (which acts like a recursive glob).
• The https://alpha.cuelang.org preprocessor uses a similar method to load a single configuration value from all the CUE files that correspond to the pages on the site (one directory of CUE files (belonging to the same package) per page).

That is to say, the current capabilities of cue/load are neither final nor complete.

This issue is a placeholder to capture experience/feedback on how cue/load should evolve.

Separately noting the related but largely orthogonal https://github.com/cue-lang/cue/issues/1410.

Describe the solution you'd like

To have native support for additional multiple loading mechanisms. This will likely include configuration such that any tooling that uses cue/load can understand and resolve CUE values from files, packages and modules in the same standard way.

Describe alternatives you've considered

For now, people are working around the lack of additional loading mechanisms with shell globs, commands etc.

(noting however that per #2000 the lack of configurability for the k8s loading mode requires more invasive work arounds).

Additional context

n/a

Please provide feedback/thoughts ideas in replies below.

[myitcv]

Related: cue/load: do not include module ancestor files in the same package by default; make optional · #2000

[alecholmez]

Big +1 for this feature request. It would greatly simplify our loading needs since we have similar use cases at greymatter.io.