eval: support late-bound references

[pgavlin]

see PR https://github.com/pulumi/esc/pull/15 for context

I have some ideas around this that I think may make this approachable.

• Let ${foo} always refer to the name that is currently in scope (i.e. defined in the current environment or one of its imports. I've been thinking of this as a "down-reference".
• Let %{foo} (or w/e sigil) always refer to the name in the the evaluation root (i.e. env in esc open env). I've been thinking of this as an "up-reference".

I am pretty sure that this eliminates any ambiguity in deciding which definition a name refers. It feels to me like an option that is not too surprising and easy to explain. Some surprises I can think of are:

• Up-references widen the set of possible circular references. For example, consider the following:

# env a
values:
  foo:
    bar: %{baz}

# env b
imports:
  - a
values:
  baz: ${foo.bar}

The reference to %{baz} will cause a cycle (a.foo.bar -> b.baz -> a.foo.bar). Without up-references, references to values in the merge stack can never be circular. With up-references they can.

• When importing an environment that does not get merged (e.g. - a: { merge: false } in an import list), any up-references in a still refer to the current evaluation root

• We might want to error when evaluating environments that have unresolved up-references (incl. for Check)

Thoughts on this?

Originally posted by @pgavlin in https://github.com/pulumi/esc/issues/15#issuecomment-1752143073

[cleverguy25]

Moving some comments here from the PR:

I wonder if differentiating will cause some confusion but it is more powerful.

If we can use any characters, I wonder if we could switch ${var} to be up references and use !{var} to be down references. Maybe this just aligns more with how I view these, where in my head I am calling down references immediate or early bound and down references as late bound, so the ! makes sense as immediate.

I do think late bound should be the default for actual interpolation vs things that reference a single symbol like aws.login probably want to be down/immediate references. But as long as there is an option I could relax on that.

[cleverguy25]

Another idea is to resolve both immediately, for use by providers or other use cases, and late bound for other "template" use cases. My PR could have been changed to resolve immediately as well pretty easily. This would at least resolve the concern here: https://github.com/pulumi/esc/pull/15#issuecomment-1740160353

[AaronFriel]

In my experience this sort of thing becomes unwieldy over time. Given the example:

# Env aws-prod
values:
  aws:
    login:
      fn::open::aws-login:
        # elided

# Env app
values:
  app:
    creds:
      fn::open::aws-secrets:
        login: ${aws.login}
        get:
          app-secret: # elided

# Env app-prod
imports:
  - app
  - aws-prod

The "aws" value becomes entangled with implicit dependencies which break if it is ever renamed, and the "app" mixin becomes hard to use in multi-account scenarios, requiring code duplication.

This is unfortunately common in Helm charts and other templating languages, where globals work up to a point, then become brittle. The end result is often that every use site (app-prod in this example) ends up inlining the complexity because it's not possible to write well behaved "functions" that take inputs and return outputs.

[cleverguy25]

My point was that providers would always be explicit as they are immediately bound. Only other simple templates would support implicit binding.

[pgavlin]

My point was that providers would always be explicit as they are immediately bound. Only other simple templates would support implicit binding.

To me this seems like it has the potential to be much more confusing: the behavior of references would silently change depending upon the context in which they appear. Then in order to understand a reference's semantics I'd need to look around it rather than at the reference alone.

[lukehoban]

We’re considering adding a notion of “context” that can be referenced - I wonder if there’s a way to leverage that for these scenarios - instead of changing the lexical scoping of direct variable references?

[pgavlin]

We’re considering adding a notion of “context” that can be referenced - I wonder if there’s a way to leverage that for these scenarios - instead of changing the lexical scoping of direct variable references?

lol, I put this together last night as a way to play around with parameterized environments: https://github.com/pulumi/esc/compare/pgavlin/context-params

that branch jams the current evaluation stack into context.values s.t. imported environments can refer to values produced by prior imports. this enables situations like this:

# Env aws-prod
values:
  aws:
    login:
      fn::open::aws-login:
        # elided

# Env app-service
values:
  service:
    name: foo

---

# Env stage
values:
  environment:
    stage: bar

---

# Env boilerplate
values:
  service-url: "https://${context.values.service.name}.${context.values.aws.region}.${context.values.environment.stage}.contoso.com"

---

# Env app-service-prod-stage
imports:
  - aws-prod
  - app-service
  - stage
  - boilerplate

It's awfully wordy, but it does get the job done.

[AaronFriel]

I like that, despite being wordy, as it doesn't close the door on naming contexts later.