String literal template escape not working `%%{`

[mackenzieATA]

Nomad version

Nomad v1.4.1 (2aa7e66bdb526e25f59883952d74dad7ea9a014e)

Operating system and Environment details

Ubuntu 20.04

Issue

Template escape sequence to allow putting a literal %{ in an argument does not work.

Reproduction steps

Run nomad agent with sudo nomad agent -dev nomad job run <jobfile.nomad> (see job file below)

Check the status of the job/allocation and see the error:

* failed to parse config:
* Invalid template control keyword: "test" is not a valid template control keyword.

Expected Result

Job runs successfully and echos %{test}

Actual Result

Job fails with:

* failed to parse config:
* Invalid template control keyword: "test" is not a valid template control keyword.

Job file (if appropriate)

job "template-test" {
  datacenters = ["dc1"]
  group "processors" {
    count = 1
    task "echotest" {
      driver = "exec"
      config {
        command = "/usr/bin/echo"
        args = [
          "%%{test}"
        ]
      }
    }
  }
}

Nomad Server logs (if appropriate)

    2022-10-19T15:05:17.390-0700 [ERROR] client.alloc_runner.task_runner: running driver failed: alloc_id=117f823f-182e-5cd7-1441-708882dd5cae task=echotest
  error=
  | 2 errors occurred:
  | \t* failed to parse config:
  | \t* Invalid template control keyword: "test" is not a valid template control keyword.
  |

    2022-10-19T15:05:17.391-0700 [INFO]  client.alloc_runner.task_runner: not restarting task: alloc_id=117f823f-182e-5cd7-1441-708882dd5cae task=echotest reason="Error was unrecoverable"

Nomad Client logs (if appropriate)

N/A

[schmichael]

Thanks for the bug report and reproduction @mackenzieATA! The problem appears to be that we decode the task.config block twice unlike every other HCL block. On the first pass %%{test} is evaluated to %{test}, then on the second pass Nomad fails to evaluate %{test}.

The second pass is performed here: https://github.com/hashicorp/nomad/blob/v1.4.1/client/allocrunner/taskrunner/task_runner.go#L874-L879

This is related to #9287 and probably other issues caused by this suboptimal handling of task.config. I think it's worth leaving this open as a distinct bug though since it might be more easily fixable than the other issue. :crossed_fingers:

Workarounds

Until we get this fixed there are a variety of workarounds:

job "rawexec-hello" {
  type        = "batch"
  datacenters = ["dc1"]

  group "rawexec-hello" {
    task "rawexec-hello" {
      driver = "raw_exec"

      config {
        command = "/bin/sh"

        # Use env vars inside the task config
        args = ["-c", "echo $NOMAD_META_testing && echo $TEST_TEMPLATE && echo $TEST_ENV"]
      }

      # Option 1: meta
      meta {
        testing = "%%{test1}"
      }

      # Option 2: env
      env {
        TEST_ENV = "%%{test2}"
      }

      # Option 3: env via templates
      template {
        destination = "local/doesnt_matter"
        env         = true
        data        = "TEST_TEMPLATE=%%{test3}"
      }
    }
  }
}

Logs:

$ nomad alloc logs 1234
%{test1}
%{test3}
%{test2}

Sorry for the hassle and thanks again for the excellent bug report.

[mackenzieATA]

Thanks for the verification and the workarounds!