buildkite
Windows build agents put nested artifacts up with slashes in file name
TLDR; the windows build agents save artifacts with what would be their nested directory structure as part of the file names. Artifacts that should be in the bin folder are named bin/foo.jpg.
Our windows build agents have two steps, one that puts up a bunch of artifacts and one that pulls down a bunch of artifacts and creates several artifacts from them.
There are other Linux build agents that also create artifacts used by the latter of those windows steps.
The first step creates artifacts as follows
artifact_paths:
- "bin*"
- "WebDisplay/bin*" (this contains 400 or so files)
- "Player/Brightsign/Web/*.xml.manifest"
there are Linux step build artifacts that also end up in the bin/ subdirectory of the repo.
If I look on s3 I'll see a bin directory with Linux build artifacts in it as well as the windows artifacts with file names like bin/foo/bar.baz IN THE ROOT directory
The second pulls all of the artifacts like this. if I don't do it exactly like this with the artifacts plugin, it will pull the files down with the slashes in their names. which is an issue when we try to change the structure of our pipelines.
plugins:
- artifacts#v1.2.0: download: [ "\*","/*" ]
Then it creates artifacts with slashes in their names in the root.
artifact_paths:
- "bin/Installers/Carousel*.exe"
- "bin/Installers/Carousel*.msi"
- "bin/Installers/BrightSign*.msi"
- "bin/Installers/InternalTools/*.msi"
- "bin/Installers/Tools/*.msi"
The long story short is, it is certainly writing the files to S3 with slashes in their file name. I tested this by writing to our private bucket and looking at the artifacts. When these artifacts are clicked on in the UI they download with underscores where the slashes would be. I've also verified that on the build agents themselves pre-upload, they are in the directory structure I would want them to be in and not created with slashes in their names.
Hi! We use the term agent. Slavery is an awful metaphor for what buildkite agents do, and also an awful metaphor for basically everything. If you could edit your issue to remove references to slavery, that would be great.
wrt. the content of your issue, can you explain a little about what your desired behaviour is, and how the existing behaviour differs from it? I'm having some trouble parsing out what the issue is here.
Hi @sethphillips — I haven't verified this, but might the normalised-upload-paths experiment help?
There's a bit more context in https://github.com/buildkite/agent/pull/1211
I came to this issue because I was facing the same thing. I have a step run on a Windows agent that uploads artifacts; then a later step on a Linux agent that tries to download the artifacts, but can't.
We are using our own artifacts S3 bucket.
On the Windows agent I tried both artifact_paths and using the artifacts#v1.9.0 plugin to perform the upload.
I tried both with / and \ as the path separator.
In all combinations the files uploaded to our S3 bucket contained the path and literal backslash characters as part of the uploaded filename (even when we specified forward slash in the upload pattern).
Then on the Linux agent I used the artifacts#v1.9.0 plugin to perform the download.
I tried both with / and \ as the path separator.
In all combinations it tried to fetch the artifacts using / as a path separator and so failed to find the file.
I can confirm that setting the normalised-upload-paths experiment on the Windows agents fixes the upload problem and the files are stored on the S3 bucket with their full paths, thus solving my problem.
