allow setting file owner/permission after artifact is downloaded

[shantanugadgil]

trafficstars

Nomad version

Nomad v0.5.6

Operating system and Environment details

CentOS 7/Ubuntu 16.04

Issue

Using this as reference: https://www.nomadproject.io/docs/job-specification/artifact.html I also tried: https://github.com/hashicorp/go-getter

How can I set file permissions to the downloaded artifact? For security reason, I would like to set 0600 on the downloaded artifact file, before my application can use it.

I hope I am missing some simple configuration option for the artifact 🙁

As an extension, is it possible to set ownership on the downloaded file. I don't need owner settings immediately, though it could come in handy later (I think)

Thanks and Regards, Shantanu

[dadgar]

@shantanugadgil No there currently isn't. I have file an issue for go-getter to add support for that. When that comes we can add it to the job file.

For now you will have to wrap your task in a script that sets the permissions and then runs the app.

[shantanugadgil]

@dadgar any updates on this?

Regards, Shantanu

[rafaelpirolla]

I had an issue trying to run tomcat inside a java driver job. As the downloaded war was inside /local/apps tomcat couldn't deploy it. I have made /local the directory and it now works but it would be great to be able to set ownership and permissions. Will follow up on the go-getter issue.

[shantanugadgil]

Any chance of this making into 0.9.0 ?

[dradtke]

For anyone else waiting on a proper fix here, my current workaround is to execute bash directly and pass it a script that sets the executable bit first:

config {
    command = "bash"
    args = ["-c", "chmod +x local/app && exec local/app"]
}

[shantanugadgil]

Nowadays what I do is put all the "shell logic" inside a script and just execute that script as the command. Usually the last line of the script is the actual app to run! :smile:

[ccakes]

I used the method suggested by @dradtke which worked fine until I upgraded to Nomad 0.9.2 yesterday and now I'm getting these in the stderr log.

chmod: changing permissions of 'local/my-bin': Operation not permitted

EDIT: I guess this is related to #5728 - using an artifact {} stanza to place the file leaves it owned by root:root so then during the job execution, nobody isn't able to chmod the file.

The nicest solution for this is for the artifact {} stanza to set permissions however a hacky solution is to cp the binary so it's owned by nobody, then chmod that...

[langmartin]

Another workaround for permissions is to package the file in a tar or zip archive with the correct permissions, and then to use the files in /local. Permissions are preserved when decompressing.

[danlsgiga]

yup, hit the same wall here. .tar.gz file is decompressed and all the files permissions are set to root.

[shantanugadgil]

The milestone changed to unscheduled 😢 ???

I know permissions do not make sense for 'all extracted file of an archive', but at least make this possible for single file downloads. For example if I am downloading a binary, I have to ensure that its executable.

If not, how about adding an example of a template block in the docs of the artifact stanza, for anyone looking for the word "perm" and/or "owner"

[tgross]

The milestone changed to unscheduled

No worries, that just means we don't have a specific release to slot this into yet, not that we're never going to tackle it.

[raypettersen]

We just hit this also.

[HarryHarcourt]

Frustratingly, also just hit this issue, file permissions not retained after decompression, I am not sure how the "exec" driver would ever be useful without the right permissions (especially as it defaults to "nobody")

[pdumon]

Same here. tarball doesn't solve: permissions are still set to root also after decompressing.

[shantanugadgil]

combined with the artifact stanza to download the tarball, I typically use the following boiler plate job script.

This allows me to then execute any chmod/chown commands as I please. 😄

job "sample" {
  datacenters = ["dc1"]

  type = "batch"

  group "sample" {
  
    task "sample" {
	
      driver = "raw_exec"

      template {
        data = <<EOH
#!/bin/bash

set -u

id -a

hostname

env | sort

sleep 30
EOH

        destination = "local/runme.bash"
      }

      config {
        command = "/bin/bash"
        args    = ["-x", "local/runme.bash"]
      }
    }
  }
}

HTH.

[pdumon]

We're using the qemu driver and artifact to specify the VM image. If I wrap the qemu commands into a raw_exec task, then I also use port_map and so forth... Tried to look into the go-getter code to see where things are going wrong but couldn't really find.

[aflatter]

Hmmm, same here. As a work-around, I prevent artifact from extracting the archive (using archive = false) and use a template to create a small wrapper bash script that extracts the archive and then runs the application.

config {
  command = "/bin/bash"
  args = ["run.sh", "--my", "--real=args"]
}

template {
  data = <<EOH
    #!/usr/bin/env bash

    cd local
    tar xzf ./release.tar.gz
    bin/myapp "$@"
  EOH

  destination = "run.sh"
  perms = "755"
}

But that's still a lot of ceremony for something really simple. I expected the artifact stanza to extract my tar archive using the same user that the task uses to run. Would that work?

[picatz]

Ran into this today while trying to create a simple Prometheus + Grafana jobspec using the exec task driver:

job "metrics" {
  datacenters = ["dc1"]
  group "prometheus_and_grafana" {
    task "prometheus" {
      driver = "exec"
      config {
        command = "./local/prometheus-2.23.0.linux-amd64/prometheus"
        args    = ["--config.file", "local/prometheus-2.23.0.linux-amd64/prometheus.yml"]
      }
      artifact {
        source = "https://github.com/prometheus/prometheus/releases/download/v2.23.0/prometheus-2.23.0.linux-amd64.tar.gz"
      }
    }
    task "grafana" {
      driver = "exec"
      config {
        command = "./local/grafana-7.0.0/bin/grafana-server"
        args    = ["-homepath", "local/grafana-7.0.0"]
      }
      artifact {
        source = "https://dl.grafana.com/oss/release/grafana-7.0.0.linux-amd64.tar.gz"
      }
    }
  }
}

Errors for the grafana task:

Failed to start grafana. error: failed to create log directory "local/grafana-7.0.0/data/log": mkdir local/grafana-7.0.0/data: permission denied

Going to try to use the mentioned workaround using a template wrapper from above.

[shantanugadgil]

closing due to age

[hashworks]

AFAIK this is still an issue? It's not possible to set permissions / owner for artifacts, they need to be provided by a script or by tarball.

[tgross]

It is, and we currently have an internal feature request around it (internal link https://hashicorp.atlassian.net/browse/FRB-384). So let's keep this open despite it's age.

[shantanugadgil]

It is, and we currently have an internal feature request around it (internal link https://hashicorp.atlassian.net/browse/FRB-384). So let's keep this open despite it's age.

for context: today I have been on a GH issue closing binge! :slightly_smiling_face:

[tgross]

That's fine and we appreciate it, but if there's anything you see the various hcc/* labels on, that usually means we've got some internal tracking on it going. Prioritization sorting is always a challenge, of course :grin:

[arodd]

Changing the owner is now supported as of 1.9.2 via #24157 with the chown artifact parameter. The ability to set numerical permissions is still being considered separately.

[lbv2rus]

Artifact ownership changing does not work with raw_exec driver in 1.9.4. With "chown = true" downloaded artifacts still are owned by root.

[joewin319]

@tgross Any update on this? or any other workaround for chmod. thanks

[tgross]

We'll generally update issues when they're being worked on.

[lbv2rus]

Artifact ownership changing WORKS with raw_exec driver in 1.9.7.