Q: How to access root-level `_ytt_lib` from inside config tree?

[mamachanko]

Libraries in _ytt_lib can only be accessed from within modules at the same directory level as _ytt_lib as far as I can tell.

However, I would love to modularize shared code into a library and share it across my deeply-nested config file-tree. I understand that this comes at a cost because now each sub-tree can't be ytt'd on its own.

Maybe this is possible already, but I can't tell how. 🤔

Flat file layout - works

Set up the kit:

.
├── _ytt_lib/
│   └── dreams/
│       └── lib.star
└── config.yaml

#! _ytt_lib/dreams/lib.star
load("@ytt:struct", "struct")

sandman = {
  "name": "The Sandman",
  "profession": "King of Dreams",
}

#! config.yaml
#@ load("@ytt:library", "library")
#@ load("@ytt:template", "template")

#@ sandman = library.get("dreams").export("sandman")
---
_: #@ template.replace(sandman)

Render it:

❯ ytt -f .
name: The Sandman
profession: King of Dreams

Nested file layout - would be nice

Set up the kit, wishfully:

.
├── _ytt_lib/
│   └── dreams/
│       └── lib.star
└── config/
    └── config.yaml

#! _ytt_lib/dreams/lib.star
load("@ytt:struct", "struct")

sandman = {
  "name": "The Sandman",
  "profession": "King of Dreams",
}

#! config/config.yaml
#@ load("@ytt:library", "library")
#@ load("@ytt:template", "template")

#@ sandman = library.get("dreams").export("sandman")
---
_: #@ template.replace(sandman)

Hope it renders, but it does not:

❯ ytt -f .
ytt: Error:
- library.get: Could not find private library (directory '_ytt_lib' missing?)
    in <toplevel>
      config/config.yaml:4 | #@ sandman = library.get("dreams").export("sandman")

---

p.s.: ❤️ ytt

[pivotaljohn]

However, I would love to modularize shared code into a library and share it across my deeply-nested config file-tree.

I think of this as the "Common Dependency" use-case. We talk about private libraries being "a dependency" for its enclosing library.

I understand that this comes at a cost because now each sub-tree can't be ytt'd on its own.

That's right: the design decision to require that all dependencies be contained within that enclosing library's directory was to attempt to keep the overall library easy to reason about (and this idea of being able to execute ytt on a dependency/library is the perfect test of that).

Maybe this is possible already, but I can't tell how. 🤔

Yes, I can think of two ways to do this.

vendir sync Common Dependencies

Perhaps the most maintainable solution I can think of that achieves that goal would be to bring vendir into the solution; it was purpose built for resolving dependencies.

Building off yer rockin' example:

#!/usr/bin/env bash
# build.sh

# refresh build/
[[ -d build ]] && rm -rf build
mkdir build
cp -R config build/

# overlay common and external dependencies
vendir sync

# vendir.yml
apiVersion: vendir.k14s.io/v1alpha1
kind: Config

# Common library references
directories:
  - path: build/config/_ytt_lib/dreams
    contents:
      - path: .
        directory:
          path: _ytt_lib/dreams

and now:

$ ytt -f build/config
name: The Sandman
profession: King of Dreams

One of the outcomes of this approach is that it treats common dependencies exactly like external dependencies. In fact, it readies a shop for not just having common dependencies within a given ytt config, but across configs (think of utility libraries used across products); all one need do is capture those common libraries in some published location (whether it be a git repo/release, imgpkg bundle, ...anywhere a vendir syncs...) and what was locally common can now be just another external dependency.

It also aligns with Carvel's adherence to the Unix Philosophy: using the right tool for the job — here, vendir resolves dependencies.

This approach naturally scales.

The trade-off is the added operations complexity:

• introducing vendir, if not already present
• requiring your ytt-templated config to be "built" in order to be used

"Common Library" Pattern

Another option (which thwarts ytt's efforts to keep libraries independent) would be to build-up an internal collection of libraries at the root of the project:

#! /common.lib.star
load("@ytt:library", "library")
load("@ytt:struct", "struct")

_dreams = library.get("dreams")

libs = struct.make(dreams=_dreams)

#! config/config.yaml
#@ load("/common.lib.star", "libs")
#@ load("@ytt:template", "template")

#@ sandman = libs.dreams.export("sandman")
---
_: #@ template.replace(sandman)

$ ytt -f .
name: The Sandman
profession: King of Dreams

This approach pragmatically absorbs the complexity within the ytt config (in the form of that /common.lib.star and all the references to it). It's appealing for its simplicity.

The trade-off is both:

• lack of the scaling qualities of the other approach, and
• diminishing ability to run/test/develop portions of your ytt config in isolation.

[mamachanko]

Thank you @pivotaljohn for describing these approaches in such detail!

Concerning "vendir sync Common Dependencies", I assume the vendir.yml would contain a path specific for each nested config-level that depends on the dependency, wouldn't there?

[pivotaljohn]

That's correct, @mamachanko.

[pivotaljohn]

Please comment on this issue if there's more to explore here.