gcs-resource icon indicating copy to clipboard operation
gcs-resource copied to clipboard

regexp not using latest version

Open kalensk opened this issue 1 year ago • 0 comments

When using regexp it does not pull the latest version of the file. Issues persists no matter if bucket versioning is enabled or not. Would it be possible to default to using the latest file version when using regexp? Or providing a new resource parameter to enable that behavior?

For example, given a resource

  - name: saved_cluster_env_files
    type: gcs
    source:
      json_key: ((gcs-service-account-key))
      bucket: some-bucket
      regexp: sub-bucket/cluster_env_files(.*).tar.gz

the first job in my pipeline pushes the resource called cluster_env_files_MyBranchName.tar.gz. Subsequent jobs then pull that same resource. However, if the pipeline get's re-flown or the first job is re-run, the following jobs are matching the correct filename, but pulling down a previous version of it. This causes those jobs to fail.

kalensk avatar May 26 '23 19:05 kalensk