spread icon indicating copy to clipboard operation
spread copied to clipboard

Artifact names are always treated as literals, missing files are ignored

Open thp-canonical opened this issue 1 year ago • 0 comments

Assuming my spread.yaml contains the following definition:

backends:
    lxd:
        systems:
            - ubuntu-22.04:
                environment:
                  PLATFORM: amd64
            - ubuntu-22.04-arm64:
                environment:
                  PLATFORM: arm64

In a task, I might want to pack a specific artifact that's named after system-specific environment variables:

# ...
execute: |
    snapcraft
    
artifacts:
    - example_1.0_${PLATFORM}.snap
# ...

Also, no wildcard expansion is possible, which could in some cases alleviate the problem of having to resort to environment variables):

# ...

artifacts:
    - example_1.0_*.snap
# ...

The problem seems to be that the list of artifacts is treated as literal string list when files are packed on the test machine:

  • The list of artifacts is read from YAML via type Task struct in spread/project.go (Artifacts [] string)
  • job.Task.Artifacts is accessed in fetchArtifacts() (spread/runner.go)
  • fetchArtifacts() calls client.RecvTar(remoteDir, job.Task.Artifacts, tarw)
  • client.RecvTar() (in spread/client.go) tries hard to treat each artifact name as literal string (including shell-quoting)
  • The tar command executed on the test machine to pack the files will be something along the lines of /bin/tar cJ --sort=name --ignore-failed-read -- 'example_1.0_${PLATFORM}.snap'

As a workaround (because client.RecvTar() uses --ignore-failed-read) one can (try to) list ALL possible values for the artifacts, and spread will ignore invalid artifacts:

# ...

artifacts:
    - example_1.0_amd64.snap
    - example_1.0_arm64.snap
# ...

Because of --ignore-failed-read, Spread won't really complain about any mistyped and unmatched file names, and silently ignore any files it cannot find. Of course, this way of maintaining artifact names is error prone and tedious, as when e.g. a new architecture is added in the future (e.g. riscv64), it's not enough to add a new system to the list of systems for the lxd backend, but every task has to have its artifact list expanded.

thp-canonical avatar Mar 13 '24 08:03 thp-canonical