[objective] Healthy Core

[viega]

We're committed to making sure our core enables developers to support our functional and non-functional requirements, in a way that is pretty easy for them.

We have built the Chalk core targeting it being:

• Highly usable, yet
• Very powerful / flexible
• Robust
• Performant

Still, there are many ways in which we can be advancing the above goals. First, key to giving us the flexibility / power while helping make it easy to deliver on usability, is our underlying configuration language con4m.

While con4m is more than good enough for our initial Chalk release, it is nowhere near what we would consider good enough for everyone else, as it mainly got enough attention to "jump the bar" for where Chalk is today, and no more.

We're not in a hurry to get con4m ready for projects outside our own, but we are committed to keeping it in good shape for chalk, and to design changes in a way that could be used by other projects from day one.

There are lots of warts we should smooth over (many of them intentional choices due to timeline), like syntax that should work better. And we focused on robustness over performance, since performance was clearly good enough for an initial release.

Just in the configuration system, there are plenty of big wins that wouldn't take too much work. For instance, most of the embedded configuration can be pre-computed with a little bit of refactoring, which would eliminate most of an (already small) startup cost.

But outside of the configuration, there are plenty such items.

For instance, there are places that, due to expediency, we used dependencies that weren't quite ideal for the use case. For instance, when doing signing of containers and other artifacts, we are currently dependent on Sigstore's cosign, which is a great tool, but is itself large and may need to be downloaded often, depending on the deployment, which isn't great when build times are crucial. Yet! Most of the core primitives to implement what we need from Sigstore are already available to us via the underlying cryptography library. We just need to implement enough pieces on top of it to provide what we need from Sigstore without being dependent on another library.

Another example where our (few) outside dependencies aren't good enough is with our Zip file handling. It currently isn't easy to do what we need to do without heavy use of tmp files, which is an issue for disk usage, among other things. We could bring in a different dependency to balance our needs better, some work required, of course.

The Zip issue overlaps with another potential concern that can impact robustness... memory footprint. In designing the initial release, we didn't worry too much about it-- virtual memory is pretty effective! But if we want to do better for more constrained environments, we can do a lot better here (for instance, even ensuring the large embedded documentation corpus lives in static memory would be a decent win if memory becomes an issue).