Don't use local s3 data for tests, use open-neurodata with `use_https=True`

[bvarjavand]

There is around 500mb of s3-formatted data in our data folder which isn't strictly necessary for testing. Specifically, the change requested is that tests for the neuron trace class use the following urls instead of searching filepaths.

"s3://open-neurodata/brainlit/brain1"
"s3://open-neurodata/brainlit/brain1_segments"

The code modified would just be these lines.

Then, we can uncomment these 2 lines in .gitignore to significantly reduce the size of the repo.

[tathey1]

let's address this first

[JacopoTeneggi]

The use of use_https=True has proven to break CloudVolume on versions of python later than 3.8.5 . Initially, we thought it was an issue limited to versions later than 3.9.0 (see #272 ), but running tests with the use of use_https=True failed (see https://github.com/neurodata/brainlit/pull/275/commits/641fa859eb79bcdc24ef1f713a318b880754a2e5 ). This is because the version of python shipped in cimg/python3.8 , which is used in CircleCI, is 3.8.9 . I tried bumping CV to the latest version, but there are some other requirements that have conflicts.

[tathey1]

So @JacopoTeneggi , is there any way for you to access our data on s3 without an access key?

[JacopoTeneggi]

Not currently @tathey1 I saved the data for tests in data/ . I can access things directly with boto, but not through cloudvolume

[tathey1]

Also, I believe use_https=True is causing issues with cloud volume in python 3.7, e.g. see NotImplementedError here

[JacopoTeneggi]

@bvarjavand , @tathey1 and I were discussing this idea: saving some service account credentials on CircleCI as environmental variables and add a step to the pipeline that will save the credentials in ./cloudvolume so that we can initialize an authenticated neuroglancer_session without worrying about use_https and delete the data from the repo

[tathey1]

@alishakodibagkar can you look into if this is possible^

[bvarjavand]

@JacopoTeneggi @tathey1 @alishakodibagkar I had things setup using the username stored in the travis.yml file, like here. The password was encrypted and only available as a travis CI environment variable, and you need both public and private keys to access the aws data. I believe the same kind of thing can be done in CircleCI according to this link. I created the .secrets file using a script here and it ran every time in the .yml file (here). Let me know if you have any questions.

[github-actions[bot]]

Stale issue message