test_get_put and test_get_put_recursive are skipped by CI

[oliverwm1]

This makes it challenging to modify code related to getting/putting files.

See:

https://github.com/dask/gcsfs/blob/60be1b9446c70277714c2ee24b9fe2be76e9c979/gcsfs/tests/test_core.py#L461

https://github.com/dask/gcsfs/blob/60be1b9446c70277714c2ee24b9fe2be76e9c979/gcsfs/tests/test_core.py#L478

and latest CI logging:

gcsfs/tests/test_core.py::test_get_put[None] SKIPPED (async fail)        [ 43%]
gcsfs/tests/test_core.py::test_get_put[size] SKIPPED (async fail)        [ 44%]
gcsfs/tests/test_core.py::test_get_put[md5] SKIPPED (async fail)         [ 45%]
gcsfs/tests/test_core.py::test_get_put[crc32c] SKIPPED (async fail)      [ 46%]
gcsfs/tests/test_core.py::test_get_put_recursive[] SKIPPED (async fail)  [ 47%]
gcsfs/tests/test_core.py::test_get_put_recursive[gs://] SKIPPED (asy...) [ 48%]
gcsfs/tests/test_core.py::test_get_put_recursive[gcs://] SKIPPED (as...) [ 49%]
gcsfs/tests/test_core.py::test_get_put_file_in_dir[] SKIPPED (async ...) [ 50%]
gcsfs/tests/test_core.py::test_get_put_file_in_dir[gs://] SKIPPED (a...) [ 50%]
gcsfs/tests/test_core.py::test_get_put_file_in_dir[gcs://] SKIPPED (...) [ 51%]

[oliverwm1]

FYI I am able to run these tests locally. Do we just need to re-record them since the async PR?

[martindurant]

I have not managed to get vcr to run these tests correctly, and I don't know why. I suspect it may be something to do with subsequent requests being too similar, and so a previous response being picked instead of the right one.

[nbren12]

Could the CI be given access to some bucket, so it could run these integration tests?

[martindurant]

Can you find an example of anywhere else that does that kind of thing? Storing a long-term token in CI secrets doesn't seem great. Better would be to figure out why this type of test confuses VCR where the rest are fine.

[nbren12]

We do this in our CI here https://github.com/VulcanClimateModeling/fv3net/blob/ea4f1b87cde9f8aa6265ad4b7e8904292bef38f3/.circleci/config.yml#L37.

The ENCODED_GOOGLE_CREDENTIALS is a secret that you can either

1. manually create and add as an env var via the web console, or
2. Use some infra-provisioning tool like terraform to setup. IMO, (1) + some button clicking in the GCP console is an easy way to get started

Storing a long-term token in CI secrets doesn't seem great

Is your concern about security? You can make a service account for just this repo and give it minimal permissions to a single bucket. You can then expire the key in the GCP console if you want to take away access at some future point.

I'm sure @frodre or I would be happy to help you set this up.