build: google cache takes too long to flush old runtime binaries

[zalberico]

• During runtime release we push a new version (e.g. 1.18) to bootstrap.urbit.org
• We then rely on next to upgrade to this version which pulls from that url
• This will fail if done quickly because google retains the old wrong version in that location in its cache
• It takes too long for google to update its storage locations

vere: next (%live): 1.18-799c519
curl: error https://bootstrap.urbit.org/vere/live/v1.18-799c519/vere-v1.18-799c519-linux-x86_64: HTTP 403
unable to save https://bootstrap.urbit.org/vere/live/v1.18-799c519/vere-v1.18-799c519-linux-x86_64 to /home/philip/tug/.bin/live/vere-v1.18-799c519-linux-x86_64: -2
vere: upgrade failed

Can we reduce the time google waits to flush the cache? Alternatively we could not overwrite and use new paths for each one (though that seems risky)

[sethfork]

It also doesn't seem like old binaries are being fully replaced / overwritten when installing per cli instructions on urbit.org

[joemfb]

I don't recall this being an issue, have the buckets been reconfigured? Alternately, our libcurl build settings have changed; maybe those need to be revisited.

[tomholford]

It's been a few hours, so either getting pretty unlucky with cache misses ... or the bucket perms need to be updated?

[zalberico]

I think part of the reason we didn't notice before may be because we're more immediately trying to upgrade after publishing. It's only an hour or so and before I think most people didn't upgrade within the hour (or day).

[zalberico]

I think the pill is different? I'm not sure we have a 1.18 pill yet, we typically do that after (couple day to week lag) to prevent accidental downgrades cc @philipcmonk

[zalberico]

From @philipcmonk

We forgot a release step: while it's fine to delay making a new pill until the next day, we need to have some pill for each binary release, otherwise nobody can boot with the new binary. i just ran gsutil cp gs://bootstrap.urbit.org/urbit-v1.1{7,8}.pill which does this.

tomorrow though we should build a real pill