allow downloading artifacts in sibling pipelines

[scriptnull]

Currently, buildkite-agent covers the following use case for downloading artifacts.

:heavy_check_mark: a step inside pipeline can download the artifacts generated by another step inside the same pipeline :heavy_check_mark: a step inside a pipeline can download the artifacts generated by another step which is present in the pipeline that triggered the current step's pipeline (parent pipeline)

We would like a step to access another step's artifact which is present in a sibling pipeline. Example: If you consider the below setup: pipeline A has two trigger steps which trigger pipeline B and pipeline C. Now the step Y can't access the artifacts present in step X because they are part of sibling pipelines.

┌───────────────────┐
│                   │
│     pipeline A    │
│                   │
└────────┬──────────┘
         │
         │
         │
         │
         │
         │                     ┌──────────────────┐
         │                     │                  │
         ├────────────────────►│ pipeline B       │
         │                     │                  │
         │                     └────────┬─────────┘
         │                              │
         │                              │                 ┌─────────────────┐
         │                              │                 │                 │
         │                              └────────────────►│    step X       │
         │                                                │                 │
         │                                                └─────────────────┘
         │
         │
         │
         │                       ┌─────────────────┐
         │                       │                 │
         └──────────────────────►│   pipeline C    │
                                 │                 │
                                 └────────┬────────┘
                                          │
                                          │
                                          │                  ┌─────────────────┐
                                          │                  │                 │
                                          └─────────────────►│   step Y        │
                                                             │                 │
                                                             └─────────────────┘

I would like to extend the artifact download command to consider those sibiling pipeline steps also while trying to download the artifacts. Let me know if this is okay to do and if it is open for contribution.

[keithduncan]

Hi @scriptnull, I’ve been looking into this and my impression is that it might already work.

The artifact download command has a --build argument which it uses to scope the artifact lookup.

Let me know if that works for you use case 😄

[scriptnull]

I am also guessing that --build would work but it makes the users go through the pain of writing some custom logic (contact buildkite API to get the build id of step X in step Y and use it along with --build argument every time. I also wonder if there is a neat way of propagating the build id of step X to step Y - let me know if you have idea about it).

Right now, whenever we run buildkite-agent artifact download [artifact path], the agent looks up at two places for the users:

• whether the artifact path is found in any of the steps in the same pipeline
• whether the artifact path is found in any of the steps in the parent pipeline

With my proposal above, I was wondering if it is possible to extend buildkite agent itself to check whether the artifact path is found in any of the sibling pipeline too. That way buildkite agent takes care of making the API calls to various different sibling pipelines and include them while checking for the artifact paths. I am not sure if this extension could be done on the agent itself or it needs changes from buildkite backend logic that looks up at various pipelines for the artifact path presence.

Maybe we introduce this with a flag like buildkite-agent artifact download [artifact path] --include-sibiling-pipleines. This will make sure the user can seamlessly use an artifact path that is part of a sibling pipeline.

(just a proposal - which would help users who are modeling their entire build pipelines as small pipelines instead of running steps directly)

We could always make those API calls ourself and use the --build argument. But I just want try my best to see if this experience could be natively improvised in the buildkite-agent itself :) so, feel free to close this issue (but implementing this proposal would add good value for our use cases and we are open to contributing too)

[keithduncan]

• whether the artifact path is found in any of the steps in the parent pipeline

I’m not actually seeing the code for this in our backend 🤔 could you share sample pipeline YAML for the triggering and triggered pipeline that demonstrates this?

I am not sure if this extension could be done on the agent itself or it needs changes from buildkite backend logic that looks up at various pipelines for the artifact path presence.

I think ultimately the change in artifact search logic to support this would have to be implemented in our agent API first 😄