References of type `plain` are not accessible during the compile time

[bdnf]

Describe the bug/feature Secrets of type plain are not accessible during the compile time in jsonnet files. Hardcoded variables from parameters are accessible as expected. Though the behaviour might be intended for refs of type AWS KMS, Vaultkv, etc. the behaviour for types plain or env should allow to access files during the compile time.

To Reproduce Steps to reproduce the behavior:

1. Assuming simple folder structure:

├── components
|   └── test-component.jsonnet
└── inventory
    └── classes
    |       └── base.yml
    └── targets
            └── microserviceyml

2. Contents of components/test-component.jsonnet:

local kap = import 'lib/kapitan.libjsonnet';
local inventory = kap.inventory();

local TOPIC = inventory.parameters.arguments.topic;
assert TOPIC == "starting_offsets.TIER1": "topic strings don't match"; # assertion fail here
assert std.asciiLower(TOPIC) == "starting_offsets.tier1": "lowercase topic strings don't match";

local database = std.asciiLower(inventory.parameters.arguments.mysql);
assert database == "some-slq-database": "strings don't match";

{
  'job': {
    apiVersion: 'batch/v1',
    kind: 'Job',
    metadata: {
      labels: {
        database: database,
        topic: std.asciiLower(TOPIC),
        deployment_name: inventory.parameters.deployment_name,
      },
      name: database + '-' + std.asciiLower(TOPIC),
    },
    spec: {}
  }
}

3. Contents of inventory/classes/base.yaml:

parameters:
  arguments:
    mysql: some-slq-DATABASE
    topic: ?{plain:topic}

  kapitan:
    compile:
      - output_path: ./${arguments:topic}/
        input_type: jsonnet
        output_type: json
        prune: True
        input_paths:
          - components/test-component.jsonnet

4. Contents of inventory/targets/microservice.yaml:

classes:
  - base

parameters:
  deployment_name: dev
  target_namespace: default

  commit_tag: 1223456789

  kapitan:
    vars:
      target: microservice

5. Set the reference of type plain:

echo -n "starting_offsets.TIER1" | kapitan refs  --write plain:topic -t  microservice -f -

6. Output is:

kapitan compile -t microservice 

Jsonnet error: failed to compile /src/kapitan/components/test-component.jsonnet:
 RUNTIME ERROR: topic strings don't match
	/src/kapitan/components/test-component.jsonnet:(5:1)-(25:2)	

Compile error: failed to compile target: microservice

Expected behavior Compiled successfully and compiled file to be compiled/microservice/starting_offsets.TIER1/job.json with contents:

{
  "apiVersion": "batch/v1",
  "kind": "Job",
  "metadata": {
    "labels": {
      "database": "some-slq-database",  # works fine
      "deployment_name": "dev",
      "topic": "starting_offsets.TIER1".  # should be lowercase
    },
    "name": "some-slq-database-starting_offsets.TIER1"  # should be lowercase
  }
}

** Possible Bug **

• kapitan --version: 0.29.5
• python --version: Python 3.7.11
• pip3 --version: pip 21.2.2 from /opt/venv/lib/python3.7/site-packages/pip (python 3.7)
• pyenv

Additional context

It would be nice to access variables that are set dynamically via plain ref type during the compile time. For example Kubernetes resources require that fields are lowercased and do not contain some special characters.

Additionally, it is not impossible to use plain:variable as a generated file name. The output file name is simple shown as:

compiled/microservice/job-?{plain:variable}.json

instead of:

compiled/microservice/job-non-secret.json

It would be nice to access them by default, or at least with some additional argument, like so:

kapitan compile -t microservice --reveal
# or
kapitan compile -t microservice --embed-refs

[deepbrook]

Just ran into this as well, and it's pretty annoying.

I was attempting to select an inventory attribute by using a value given via env like this:

# kapitan inventory class file
---
parameters:
  target_team_cn: "ug_?{env:env_vars/team}"
...
# Jinja template
{{ inventory.parameters.teams[inventory.parameters.target_team_cn] }}

But it fails because the env value isn't resolved at compile time. Naturally, using --reveal does not work either.

..devel/deploy on  refac-purpose-config +1539 -596 [$✘»!+?] w 🐍 v3.8.10 (purpose-configuration) ☁️ default
🐟 ❯ kapitan compile
Rendered inventory (0.02s)

Jinja2 error: failed to render /home/deepbrook/devel/deploy/templates/team.tfvars: Jinja2 TemplateError: 'dict object' has no attribute 'ug_?{env:env_vars/team}', at /home/deepbrook/devel/deploy/templates/team.tfvars:8
Compile error: failed to compile target: team

This essentially forces me to do either of the following:

• write 43 target files (one for each of our teams)
• render the above file for each team every single time for the team

Since we also have secrets in these rendered templates, the latter solution isn't viable.

🐟 ❯ kapitan --version
0.31.0