[batch] Use sops for credentials

[lgruen]

Context: https://hail.zulipchat.com/#narrow/stream/300487-Hail-Batch-Dev/topic/ci_config.20secrets

#assign services

[danking]

@daniel-goldstein @jigold let's talk about this today/tomorrow? This seems like an improvement to me.

[lgruen]

Friendly ping, as merging this would help keep our fork in sync.

[jigold]

@daniel-goldstein I think I am fine with this change in theory. However, it is unclear to me how this would work if we added our encoded files to the repository. Would other forks have overwrite ours each time they merged changes? Would you want me to follow the new directions in my hail-jigold test project to verify the new changes are good?

[daniel-goldstein]

Forks would indeed need to overwrite ours, but since the file wouldn't change much it seems like that's not much of a hassle to maintain, right Leo? And ya this seems like a fine change but we would need to follow up right afterward with our own credentials.

[lgruen]

Forks would indeed need to overwrite ours, but since the file wouldn't change much it seems like that's not much of a hassle to maintain, right Leo? And ya this seems like a fine change but we would need to follow up right afterward with our own credentials.

Yes, alternatively we could also use the GitHub organization name or something similar when constructing the file path to encrypted credentials, to avoid collisions completely. (Forks like the CPG one would only add files to their deployments.)

[jigold]

I think if it's not too hard of a change, we should add the files with encoded secrets to something like infra/gcp/data/.... This makes it clear that these files have a different purpose and gives some indication that they're specific to your repo. If you want to also add prefixing the file name with the repo, then that would make it even clearer. But if it's too much work, don't bother. Maybe something like infra/gcp/data/hail-is/ etc.

[lgruen]

I think if it's not too hard of a change, we should add the files with encoded secrets to something like infra/gcp/data/.... This makes it clear that these files have a different purpose and gives some indication that they're specific to your repo. If you want to also add prefixing the file name with the repo, then that would make it even clearer. But if it's too much work, don't bother. Maybe something like infra/gcp/data/hail-is/ etc.

I've put all deployment-specific configs in an $ORGANIZATION_DOMAIN subfolder now, which hopefully should avoid any collisions.

[lgruen]

Just realized that this is still open. Are there any missing changes?

[jigold]

I'm so sorry. I didn't see you addressed my comments about the domain. I'll double check it one last time, but I'm sure it's fine.