gcs-resource
gcs-resource copied to clipboard
regexp not using latest version
When using regexp it does not pull the latest version of the file. Issues persists no matter if bucket versioning is enabled or not. Would it be possible to default to using the latest file version when using regexp? Or providing a new resource parameter to enable that behavior?
For example, given a resource
- name: saved_cluster_env_files
type: gcs
source:
json_key: ((gcs-service-account-key))
bucket: some-bucket
regexp: sub-bucket/cluster_env_files(.*).tar.gz
the first job in my pipeline pushes the resource called cluster_env_files_MyBranchName.tar.gz
. Subsequent jobs then pull that same resource. However, if the pipeline get's re-flown or the first job is re-run, the following jobs are matching the correct filename, but pulling down a previous version of it. This causes those jobs to fail.